ci: Print out how long each step takes on CI
This commit updates CI configuration to inform rustbuild that it should print
out how long each step takes on CI. This'll hopefully allow us to track the
duration of steps over time and follow regressions a bit more closesly (as well
as have closer analysis of differences between two builds).
cc #48829
This commit updates CI configuration to inform rustbuild that it should print
out how long each step takes on CI. This'll hopefully allow us to track the
duration of steps over time and follow regressions a bit more closesly (as well
as have closer analysis of differences between two builds).
cc #48829
Faster submodule updating
For the common case when there are no submodules which need updating, this takes 0.48 seconds instead of 47 seconds.
r? @alexcrichton
Support extra-verbose builds
- The bootstrap crate currently passes -v to Cargo if itself invoked with -vv. But Cargo supports -vv (to show build script output), so make bootstrap pass that if itself invoked with -vvv. (More specifically, pass N '-v's to Cargo if invoked with N+1 of them.)
- bootstrap.py currently tries to pass on up to two '-v's to cargo when building bootstrap, but incorrectly ('-v' is marked as 'store_true', so argparse stores either False or True, ignoring multiple '-v's). Fix this, allow passing any number of '-v's, and make it consistent with bootstrap's invocation of Cargo (i.e. subtract one from the number of '-v's).
- Also improve bootstrap.py's config.toml 'parsing' to support arbitrary verbosity levels, + allow command line to override it.
rustc: Enable embedding LLVM bitcode for iOS
This commit updates rustc to embed bitcode in each object file generated by
default when compiling for iOS. This was determined in #35968 as a step
towards better compatibility with the iOS toolchain, so let's give it a spin and
see how it turns out!
Note that this also updates the `cc` dependency which should propagate this
change of embedding bitcode for C dependencies as well.
- The bootstrap crate currently passes -v to Cargo if itself invoked
with -vv. But Cargo supports -vv (to show build script output), so make
bootstrap pass that if itself invoked with -vvv. (More specifically,
pass N '-v's to Cargo if invoked with N+1 of them.)
- bootstrap.py currently tries to pass on up to two '-v's to cargo when
building bootstrap, but incorrectly ('-v' is marked as 'store_true', so
argparse stores either False or True, ignoring multiple '-v's). Fix
this, allow passing any number of '-v's, and make it consistent with
bootstrap's invocation of Cargo (i.e. subtract one from the number of
'-v's).
- Also improve bootstrap.py's config.toml 'parsing' to support arbitrary
verbosity levels, + allow command line to override it.
rustbuild: Remove ThinLTO-related configuration
This commit removes some ThinLTO/codegen unit cruft primarily only needed during
the initial phase where we were adding ThinLTO support to rustc itself. The
current bootstrap compiler knows about ThinLTO and has it enabled by default for
multi-CGU builds which are also enabled by default. One CGU builds (aka
disabling ThinLTO) can be achieved by configuring the number of codegen units to
1 for a particular builds.
This also changes the defaults for our dist builders to go back to multiple
CGUs. Unfortunately we're seriously bleeding for cycle time on the bots right
now so we need to recover any time we can.
* Pass `opt_level(2)` when calculating CFLAGS to get the right flags on iOS
* Unconditionally pass `-O2` when compiling libbacktrace
This should...
Close#48903Close#48906
Required moving all fulldeps tests depending on `rand` to different locations as
now there's multiple `rand` crates that can't be implicitly linked against.
Remove ONLY_BUILD and ONLY_BUILD_TARGETS
Primarily removes `ONLY_BUILD` and `ONLY_BUILD_TARGETS`. These aren't actually needed in the new system since we can simply not take the relevant `host` and `target` fields if we don't want to run with them in `Step::make_run`.
This PR also includes a few other commits which generally clean up the state of rustbuild, but are not related to the `Step` changes.
We explicitly do this when compiling OpenSSL itself due to weird racy issues in
its build system, and now we've started seeing issues in the `make install` step
so let's try and see what ratcheting down the parallelism does here...
travis: Upgrade dist builders for OSX
This commit upgrades the dist builders for OSX to Travis's new `xcode9.3-moar`
image which has 3 cores available to it instead of 2. This should help us
provide speedier builds on OSX and hit timeouts less in theory!
Note that historically the dist builders for OSX have been a different version
than the ones that are running tests. I had forgotten why this was the case and
digging around brought up 307615567 where apparently Xcode 8 wasn't able to
compile LLVM with `MACOSX_DEPLOYMENT_TARGET=10.7` which we desired. On a whim I
gave this PR a spin and it [looks like][green] this has since been fixed (maybe
in LLVM?). In any case those green builds should hopefully mean that we can
safely upgrade and get faster infrastructure to boot.
This commit also includes an upgrade of OpenSSL. This is not done for security
reasons but rather build system reasons. Originally builds with the new image
[did not succeed][red] due to weird build failures in OpenSSL, but upgrading
seems to have made the spurious errors go away to here's to also hoping that's
fixed!
[green]: https://travis-ci.org/rust-lang/rust/builds/351353412
[red]: https://travis-ci.org/rust-lang/rust/builds/350969248
This commit removes some ThinLTO/codegen unit cruft primarily only needed during
the initial phase where we were adding ThinLTO support to rustc itself. The
current bootstrap compiler knows about ThinLTO and has it enabled by default for
multi-CGU builds which are also enabled by default. One CGU builds (aka
disabling ThinLTO) can be achieved by configuring the number of codegen units to
1 for a particular builds.
This also changes the defaults for our dist builders to go back to multiple
CGUs. Unfortunately we're seriously bleeding for cycle time on the bots right
now so we need to recover any time we can.
This commit upgrades the dist builders for OSX to Travis's new `xcode9.3-moar`
image which has 3 cores available to it instead of 2. This should help us
provide speedier builds on OSX and hit timeouts less in theory!
Note that historically the dist builders for OSX have been a different version
than the ones that are running tests. I had forgotten why this was the case and
digging around brought up 307615567 where apparently Xcode 8 wasn't able to
compile LLVM with `MACOSX_DEPLOYMENT_TARGET=10.7` which we desired. On a whim I
gave this PR a spin and it [looks like][green] this has since been fixed (maybe
in LLVM?). In any case those green builds should hopefully mean that we can
safely upgrade and get faster infrastructure to boot.
This commit also includes an upgrade of OpenSSL. This is not done for security
reasons but rather build system reasons. Originally builds with the new image
[did not succeed][red] due to weird build failures in OpenSSL, but upgrading
seems to have made the spurious errors go away to here's to also hoping that's
fixed!
[green]: https://travis-ci.org/rust-lang/rust/builds/351353412
[red]: https://travis-ci.org/rust-lang/rust/builds/350969248
For LLD integration the path to `llvm-config` needed to change to inside the
build directory itself (for whatever reason) but the build directory is
different on MSBuild than it is on `ninja` for MSVC builds, so the path to
`llvm-config.exe` was actually wrong and not working!
This commit removes the `Build::llvm_config` function in favor of the source of
truth, the `Llvm` build step itself. The build step was then updated to find the
right build directory for MSBuild as well as `ninja` for where `llvm-config.exe`
is located.
Closes#48749
rustbuild: Pass `ccache` to build scripts
This is a re-attempt at #48192 hopefully this time with 100% less randomly
[blocking builds for 20 minutes][block]. To work around #48192 the sccache
server is started in the `run.sh` script very early on in the compilation
process.
[block]: https://github.com/rust-lang/rust/issues/48192
This commit refactors how the path to the linker that we're going to invoke is
selected. Previously all targets listed *both* a `LinkerFlavor` and a `linker`
(path) option, but this meant that whenever you changed one you had to change
the other. The purpose of this commit is to avoid coupling these where possible.
Target specifications now only unconditionally define the *flavor* of the linker
that they're using by default. If not otherwise specified each flavor now
implies a particular default linker to run. As a result, this means that if
you'd like to test out `ld` for example you should be able to do:
rustc -Z linker-flavor=ld foo.rs
whereas previously you had to do
rustc -Z linker-flavor=ld -C linker=ld foo.rs
This will hopefully make it a bit easier to tinker around with variants that
should otherwise be well known to work, for example with LLD, `ld` on OSX, etc.
This commit imports the LLD project from LLVM to serve as the default linker for
the `wasm32-unknown-unknown` target. The `binaryen` submoule is consequently
removed along with "binaryen linker" support in rustc.
Moving to LLD brings with it a number of benefits for wasm code:
* LLD is itself an actual linker, so there's no need to compile all wasm code
with LTO any more. As a result builds should be *much* speedier as LTO is no
longer forcibly enabled for all builds of the wasm target.
* LLD is quickly becoming an "official solution" for linking wasm code together.
This, I believe at least, is intended to be the main supported linker for
native code and wasm moving forward. Picking up support early on should help
ensure that we can help LLD identify bugs and otherwise prove that it works
great for all our use cases!
* Improvements to the wasm toolchain are currently primarily focused around LLVM
and LLD (from what I can tell at least), so it's in general much better to be
on this bandwagon for bugfixes and new features.
* Historical "hacks" like `wasm-gc` will soon no longer be necessary, LLD
will [natively implement][gc] `--gc-sections` (better than `wasm-gc`!) which
means a postprocessor is no longer needed to show off Rust's "small wasm
binary size".
LLD is added in a pretty standard way to rustc right now. A new rustbuild target
was defined for building LLD, and this is executed when a compiler's sysroot is
being assembled. LLD is compiled against the LLVM that we've got in tree, which
means we're currently on the `release_60` branch, but this may get upgraded in
the near future!
LLD is placed into rustc's sysroot in a `bin` directory. This is similar to
where `gcc.exe` can be found on Windows. This directory is automatically added
to `PATH` whenever rustc executes the linker, allowing us to define a `WasmLd`
linker which implements the interface that `wasm-ld`, LLD's frontend, expects.
Like Emscripten the LLD target is currently only enabled for Tier 1 platforms,
notably OSX/Windows/Linux, and will need to be installed manually for compiling
to wasm on other platforms. LLD is by default turned off in rustbuild, and
requires a `config.toml` option to be enabled to turn it on.
Finally the unstable `#![wasm_import_memory]` attribute was also removed as LLD
has a native option for controlling this.
[gc]: https://reviews.llvm.org/D42511
Don't produce TOCs for doc markdown files
Currently, we are producing headers for markdown files,
which is generally not what we want. As such, passing this
flag causes them to render normally.
https://doc.rust-lang.org/nightly/book/ is an example page currently where this is done incorrectly.
Restore the download of rust-mingw
The build might otherwise break due to mixing MinGW object files from rust-std and the local MinGW which might be newer/older than the version used to build rust-std.
Fixes#48272
r? @alexcrichton
Improve --help performance for x.py
Since compiling the bootstrap command doesn't require any submodules,
we can skip updating submodules when a --help command is passed in.
On my machine, this saves 1 minute if the submodules are already
downloaded, and 10 minutes if run on a clean repo.
This commit also adds a message before compiling/downloading anything
when a --help command is passed in, to tell the user WHY --help
takes so long to complete. It also points the user to the bootstrap
README.md for faster help.
Finally, this fixes one warning message that still referenced using
make instead of x.py, even though x.py is now the standard way of
building rust.
Closes#37305
Currently, we are producing headers for markdown files,
which is generally not what we want. As such, passing this
flag causes them to render normally.
It was an existing solution to tell the user why a --help command
takes a long time to process. However, it would only print if the
stage0 rust compiler needed to be downloaded, it came after
update_submodules (which took a long time), and it was immediately
followed by download messages and loading bars, meaning users could
easily gloss over the message.
This commit also moves the help message out of main(), and instead
puts it at the top of bootstrap(). main() is intended to be minimal,
only handling error messages.
This is a re-attempt at #48192 hopefully this time with 100% less randomly
[blocking builds for 20 minutes][block]. To work around #48192 the sccache
server is started in the `run.sh` script very early on in the compilation
process.
[block]: https://github.com/rust-lang/rust/issues/48192
The build might otherwise break due to mixing MinGW object files from
rust-std and the local MinGW which might be newer/older than the version
used to build rust-std.
Fixes#48272
Fix error-format to properly send JSON to stdout
Since we take Cargo's JSON messages as well we need to specifically send
rustc's messages out so we don't hide them.
r? @Manishearth
