This commit updates the `ToolBuild` step to stream Cargo's JSON messages, parse
them, and record all libraries built. If we build anything twice (aka Cargo)
it'll most likely happen due to dependencies being recompiled which is caught by
this check.
Support extra-verbose builds
- The bootstrap crate currently passes -v to Cargo if itself invoked with -vv. But Cargo supports -vv (to show build script output), so make bootstrap pass that if itself invoked with -vvv. (More specifically, pass N '-v's to Cargo if invoked with N+1 of them.)
- bootstrap.py currently tries to pass on up to two '-v's to cargo when building bootstrap, but incorrectly ('-v' is marked as 'store_true', so argparse stores either False or True, ignoring multiple '-v's). Fix this, allow passing any number of '-v's, and make it consistent with bootstrap's invocation of Cargo (i.e. subtract one from the number of '-v's).
- Also improve bootstrap.py's config.toml 'parsing' to support arbitrary verbosity levels, + allow command line to override it.
- The bootstrap crate currently passes -v to Cargo if itself invoked
with -vv. But Cargo supports -vv (to show build script output), so make
bootstrap pass that if itself invoked with -vvv. (More specifically,
pass N '-v's to Cargo if invoked with N+1 of them.)
- bootstrap.py currently tries to pass on up to two '-v's to cargo when
building bootstrap, but incorrectly ('-v' is marked as 'store_true', so
argparse stores either False or True, ignoring multiple '-v's). Fix
this, allow passing any number of '-v's, and make it consistent with
bootstrap's invocation of Cargo (i.e. subtract one from the number of
'-v's).
- Also improve bootstrap.py's config.toml 'parsing' to support arbitrary
verbosity levels, + allow command line to override it.
Remove ONLY_BUILD and ONLY_BUILD_TARGETS
Primarily removes `ONLY_BUILD` and `ONLY_BUILD_TARGETS`. These aren't actually needed in the new system since we can simply not take the relevant `host` and `target` fields if we don't want to run with them in `Step::make_run`.
This PR also includes a few other commits which generally clean up the state of rustbuild, but are not related to the `Step` changes.
For LLD integration the path to `llvm-config` needed to change to inside the
build directory itself (for whatever reason) but the build directory is
different on MSBuild than it is on `ninja` for MSVC builds, so the path to
`llvm-config.exe` was actually wrong and not working!
This commit removes the `Build::llvm_config` function in favor of the source of
truth, the `Llvm` build step itself. The build step was then updated to find the
right build directory for MSBuild as well as `ninja` for where `llvm-config.exe`
is located.
Closes#48749
This commit imports the LLD project from LLVM to serve as the default linker for
the `wasm32-unknown-unknown` target. The `binaryen` submoule is consequently
removed along with "binaryen linker" support in rustc.
Moving to LLD brings with it a number of benefits for wasm code:
* LLD is itself an actual linker, so there's no need to compile all wasm code
with LTO any more. As a result builds should be *much* speedier as LTO is no
longer forcibly enabled for all builds of the wasm target.
* LLD is quickly becoming an "official solution" for linking wasm code together.
This, I believe at least, is intended to be the main supported linker for
native code and wasm moving forward. Picking up support early on should help
ensure that we can help LLD identify bugs and otherwise prove that it works
great for all our use cases!
* Improvements to the wasm toolchain are currently primarily focused around LLVM
and LLD (from what I can tell at least), so it's in general much better to be
on this bandwagon for bugfixes and new features.
* Historical "hacks" like `wasm-gc` will soon no longer be necessary, LLD
will [natively implement][gc] `--gc-sections` (better than `wasm-gc`!) which
means a postprocessor is no longer needed to show off Rust's "small wasm
binary size".
LLD is added in a pretty standard way to rustc right now. A new rustbuild target
was defined for building LLD, and this is executed when a compiler's sysroot is
being assembled. LLD is compiled against the LLVM that we've got in tree, which
means we're currently on the `release_60` branch, but this may get upgraded in
the near future!
LLD is placed into rustc's sysroot in a `bin` directory. This is similar to
where `gcc.exe` can be found on Windows. This directory is automatically added
to `PATH` whenever rustc executes the linker, allowing us to define a `WasmLd`
linker which implements the interface that `wasm-ld`, LLD's frontend, expects.
Like Emscripten the LLD target is currently only enabled for Tier 1 platforms,
notably OSX/Windows/Linux, and will need to be installed manually for compiling
to wasm on other platforms. LLD is by default turned off in rustbuild, and
requires a `config.toml` option to be enabled to turn it on.
Finally the unstable `#![wasm_import_memory]` attribute was also removed as LLD
has a native option for controlling this.
[gc]: https://reviews.llvm.org/D42511
Previously, a Step would be able to tell on its own when it was invoked
"by-default" (that is, `./x.py test` was called instead of `./x.py test
some/path`). This commit replaces that functionality, invoking each Step
with each of the paths it has specified as "should be invoked by."
For example, if a step calls `path("src/tools/cargo")` and
`path("src/doc/cargo")` then it's make_run will be called twice, with
"src/tools/cargo" and "src/doc/cargo." This makes it so that default
handling logic is in builder, instead of spread across various Steps.
However, this meant that some Step specifications needed to be updated,
since for example `rustdoc` can be built by `./x.py build
src/librustdoc` or `./x.py build src/tools/rustdoc`. A `PathSet`
abstraction is added that handles this: now, each Step can not only list
`path(...)` but also `paths(&[a, b, ...])` which will make it so that we
don't invoke it with each of the individual paths, instead invoking it
with the first path in the list (though this shouldn't be depended on).
Future work likely consists of implementing a better/easier way for a
given Step to work with "any" crate in-tree, especially those that want
to run tests, build, or check crates in the std, test, or rustc crate
trees. Currently this is rather painful to do as most of the logic is
duplicated across should_run and make_run. It seems likely this can be
abstracted away into builder somehow.
This commit introduces a separately compiled backend for Emscripten, avoiding
compiling the `JSBackend` target in the main LLVM codegen backend. This builds
on the foundation provided by #47671 to create a new codegen backend dedicated
solely to Emscripten, removing the `JSBackend` of the main codegen backend in
the process.
A new field was added to each target for this commit which specifies the backend
to use for translation, the default being `llvm` which is the main backend that
we use. The Emscripten targets specify an `emscripten` backend instead of the
main `llvm` one.
There's a whole bunch of consequences of this change, but I'll try to enumerate
them here:
* A *second* LLVM submodule was added in this commit. The main LLVM submodule
will soon start to drift from the Emscripten submodule, but currently they're
both at the same revision.
* Logic was added to rustbuild to *not* build the Emscripten backend by default.
This is gated behind a `--enable-emscripten` flag to the configure script. By
default users should neither check out the emscripten submodule nor compile
it.
* The `init_repo.sh` script was updated to fetch the Emscripten submodule from
GitHub the same way we do the main LLVM submodule (a tarball fetch).
* The Emscripten backend, turned off by default, is still turned on for a number
of targets on CI. We'll only be shipping an Emscripten backend with Tier 1
platforms, though. All cross-compiled platforms will not be receiving an
Emscripten backend yet.
This commit means that when you download the `rustc` package in Rustup for Tier
1 platforms you'll be receiving two trans backends, one for Emscripten and one
that's the general LLVM backend. If you never compile for Emscripten you'll
never use the Emscripten backend, so we may update this one day to only download
the Emscripten backend when you add the Emscripten target. For now though it's
just an extra 10MB gzip'd.
Closes#46819
Building on the work of # 45684 this commit updates the compiler to
unconditionally load the `rustc_trans` crate at runtime instead of linking to it
at compile time. The end goal of this work is to implement # 46819 where rustc
will have multiple backends available to it to load.
This commit starts off by removing the `extern crate rustc_trans` from the
driver. This involved moving some miscellaneous functionality into the
`TransCrate` trait and also required an implementation of how to locate and load
the trans backend. This ended up being a little tricky because the sysroot isn't
always the right location (for example `--sysroot` arguments) so some extra code
was added as well to probe a directory relative to the current dll (the
rustc_driver dll).
Rustbuild has been updated accordingly as well to have a separate compilation
invocation for the `rustc_trans` crate and assembly it accordingly into the
sysroot. Finally, the distribution logic for the `rustc` package was also
updated to slurp up the trans backends folder.
A number of assorted fallout changes were included here as well to ensure tests
pass and such, and they should all be commented inline.
We can't use git commands to compute a prerelease version when we're
building from a source tarball, or if git is otherwise unavailable.
We'll just call such builds `x.y.z-beta`, without a prerelease.
This currently only supports a limited subset of the full compilation,
but is likely 90% of what people will want and is possible without
building a full compiler (i.e., running LLVM). In theory, this means
that contributors who don't want to build LLVM now have an easy way to
compile locally, though running tests won't work.
This is a forward-port of:
* 9426dda83d7a928d6ced377345e14b84b0f11c21
* cbfb9858951da7aee22d82178405306fca9decb1
from the beta branch which is used to automatically calculate the beta number
based on the number of merges to the beta branch so far.
This reverts commit ab018c76e1.
This also adds the `ToolBuild::is_ext_tool` field to replace the previous
`ToolBuild::expectation` field, to indicate whether a build-failure of
certain tool is essential.
We copy built tool binaries into a dedicated directory to avoid deleting
them, stageN-tools-bin. These aren't ever cleared out by code, since
there should be no reason to do so, and we'll simply overwrite them as
necessary.
When clearing out the stageN-{std,rustc,tools} directories, make sure to
delete both Cargo directories -- per-target and build scripts. This
ensures that changing libstd doesn't cause problems due to build scripts
not being rebuilt, even though they should be.
rustbuild: Avoid some extraneous rustc compiles on cross builds
This tweaks a few locations here and there to avoid compiling rustc too many times on our cross-builders on CI.
Closes https://github.com/rust-lang/rust/issues/44132
This controls the value of the crt-static feature used when building the
standard library for a target, as well as the compiler itself when that
target is the host.
Some users of the build system change the git sha on every build due to
utilizing git to push changes to a remote server. This allows them to
simply configure that away instead of depending on custom patches to
rustbuild.
This introduces a slight change in behavior, where we unilaterally
respect the --host and --target parameters passed for all sanity
checking and runtime configuration.
