rustbuild: Add sccache support
This commit adds support for sccache, a ccache-like compiler which works on MSVC
and stores results into an S3 bucket. This also switches over all Travis and
AppVeyor automation to using sccache to ensure a shared and unified cache over
time which can be shared across builders.
The support for sccache manifests as a new `--enable-sccache` option which
instructs us to configure LLVM differently to use a 'sccache' binary instead of
a 'ccache' binary. All docker images for Travis builds are updated to download
Mozilla's tooltool builds of sccache onto various containers and systems.
Additionally a new `rust-lang-ci-sccache` bucket is configured to hold all of
our ccache goodies.
---
Note that this does not currently change Windows [due to previously written up issues](https://github.com/rust-lang/rust/issues/38119#issuecomment-266631585). Despite that, however, I was curious to get timings for the builds on Travis to see what ranges we're working with. As a result, this is a WIP PR I'm using to gauge build times and such.
rustbuild: Package rust-mingw by default
This fixes the `make dist` step on MinGW to package the `rust-mingw` component
by default. This should hopefully be the last step in fixing nightlies.
rustbuild: Add cli option --keep-stage
This option is intended to be used like:
./x.py build --stage 1 --keep-stage 0
Which skips all stage 0 steps, so that stage 1 can be recompiled
directly (even if for example libcore has changes).
This is useful when working on `cfg(not(stage0))` parts of the
libraries or when re-running stage 1 tests in libraries in general.
Fixes#38326
This commit adds support for sccache, a ccache-like compiler which works on MSVC
and stores results into an S3 bucket. This also switches over all Travis and
AppVeyor automation to using sccache to ensure a shared and unified cache over
time which can be shared across builders.
The support for sccache manifests as a new `--enable-sccache` option which
instructs us to configure LLVM differently to use a 'sccache' binary instead of
a 'ccache' binary. All docker images for Travis builds are updated to download
Mozilla's tooltool builds of sccache onto various containers and systems.
Additionally a new `rust-lang-ci-sccache` bucket is configured to hold all of
our ccache goodies.
Document --test-args for rustbuild
There are three changes:
* Replace --filter with --test-args
* Delete `./x.py test src/test/run-pass/assert-*` example, which doesn't work
* As driveby, update Buildbot URLs to https
Fix#38275.
r? @alexcrichton
This commit skips a few more dist tragets during compilation which shouldn't be
necessary.
* First, when packaging std we only take action when the host target is the
build target. Otherwise we package the same artifacts a number of times, which
shouldn't be necessary.
* Next, we apply the same logic to the save-analysis build. This is actually
required for correctness as the build compiler is the only one which actually
has save analysis information. This should fix an error seen on nightlies.
This option is intended to be used like:
./x.py build --stage 1 --keep-stage 0
Which skips all stage 0 steps, so that stage 1 can be recompiled
directly (even if for example libcore has changes).
This is useful when working on `cfg(not(stage0))` parts of the
libraries, or when re-running stage 1 tests in libraries in general.
Git worktrees have this as a file and typically won't work inside docker
containers, but that's ok, so instead of just checking for existence check for a
directory to see if the git commands will succeed.
Make configure message re x.py not assume build dir == src dir
Fix#38251 but perhaps not BEST fix for it.
As driveby, fix copyright year in `Makefile.in`
Prevent Windows from displaying UI on errors.
Otherwise tests like run-pass/out-of-stack get wedged on Windows error reporting dialog (unless error reporting has been disabled, of course).
This commit implements the `distcheck` target for rustbuild which is only ever
run on our nightly bots. This essentially just creates a tarball, un-tars it,
and then runs a full build, validating that the release tarballs do indeed have
everything they need to build Rust.
rustbuild: Use src/rustc for assembled compilers
The `src/rustc` path is intended for assembling a compiler (e.g. the bare bones)
not actually compiling the whole compiler itself. This path was accidentally
getting hijacked to represent the whole compiler being compiled, so let's
redirect that elsewhere for that particular cargo project.
Closes#38039
This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2016-01-02.
[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368
This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.
Closes#37858
The `src/rustc` path is intended for assembling a compiler (e.g. the bare bones)
not actually compiling the whole compiler itself. This path was accidentally
getting hijacked to represent the whole compiler being compiled, so let's
redirect that elsewhere for that particular cargo project.
Closes#38039
Add rustbuild command `bench`
Add command bench to rustbuild, so that `./x.py bench <path>` can compile and run benchmarks.
`./x.py bench --stage 1 src/libcollections` and `./x.py bench --stage 1 src/libstd` should both compile well. Just `./x.py bench` runs all benchmarks for the libstd crates.
Fixes#37897
with this feature disabled, you can (Cargo) compile std with
"panic=abort"
rustbuild will build std with this feature enabled, to maintain the
status quo
fixes#37252
rustbuild: allow dynamically linking LLVM
The makefiles and `mklldeps.py` called `llvm-config --shared-mode` to
find out if LLVM defaulted to shared or static libraries, and just went
with that. But under rustbuild, `librustc_llvm/build.rs` was assuming
that LLVM should be static, and even forcing `--link-static` for 3.9+.
Now that build script also uses `--shared-mode` to learn the default,
which should work better for pre-3.9 configured for dynamic linking, as
it wasn't possible back then to choose differently via `llvm-config`.
Further, the configure script now has a new `--enable-llvm-link-shared`
option, which allows one to manually override `--link-shared` on 3.9+
instead of forcing static.
Update: There are now four static/shared scenarios that can happen
for the supported LLVM versions:
- 3.9+: By default use `llvm-config --link-static`
- 3.9+ and `--enable-llvm-link-shared`: Use `--link-shared` instead.
- 3.8: Use `llvm-config --shared-mode` and go with its answer.
- 3.7: Just assume static, maintaining the status quo.
rustbuild: Allow configuration of python interpreter
Add a configuration key to `config.toml`, read it from `./configure`, and add
auto-detection if none of those were specified.
Closes#35760
The makefiles and `mklldeps.py` called `llvm-config --shared-mode` to
find out if LLVM defaulted to shared or static libraries, and just went
with that. But under rustbuild, `librustc_llvm/build.rs` was assuming
that LLVM should be static, and even forcing `--link-static` for 3.9+.
Now that build script also uses `--shared-mode` to learn the default,
which should work better for pre-3.9 configured for dynamic linking, as
it wasn't possible back then to choose differently via `llvm-config`.
Further, the configure script now has a new `--enable-llvm-link-shared`
option, which allows one to manually override `--link-shared` on 3.9+
instead of forcing static.
Add llvm debuginfo configure option
CC @nnethercote @Mark-Simulacrum
We add a new configure option, `--enable-llvm-debuginfo`, to do exactly what you'd think.
Re: #31033Fixes#37738
If a rule is flagged with `default(true)` then the pseudo-rule `default:foo`
will include that. If a rule is also flagged with `.host(true)`, however, then
the rule shouldn't be included for targets that aren't in the host array. This
adds a filter to ensure we don't pull in host rules for targets by accident.
update_with_config_mk() needs to read the new llvm debuginfo config
option from config.mk. Other than that, rustbuild already supports
LLVM's RelWithDebInfo build type.
compile-test: allow overriding nodejs binary location
Add a command-line argument to manually specify which nodejs binary should be used,
which disables the default search.
Original work done by @tari.
Fixes#34188.
rustbuild: enable an initial local cargo
This allows the initial build of src/bootstrap itself to use a local
cargo taken from `configure --local-rust-root`. It was already finding
rustc this way, but was always downloading cargo since it didn't know
where to find it.
It now matches the same logic that `config.rs` will use for stage0,
where both rustc and cargo are taken from `CFG_LOCAL_RUST_ROOT`.
Move all Linux/OSX CI infastructure to Travis
This commit configures our `.travis.yml` to test the full suite of tests we have
on Buildbot right now. A whole mess of docker images are added to the `src/ci`
directory which represent all the build environments for each configuration.
Each of these environments is then configured in `.travis.yml` to run on the
auto branch.
Note that the full matrix of tests aren't intended to be run on all PRs.
Instead, we continue to run only one entry in the matrix, forcing all others to
finish quickly. Only the `auto` branch should run the full matrix of builds.
Also note that the infrastructure hasn't quite been allocated yet to the
rust-lang/rust repository, so everything is disabled for now except for the one
build that happens on PRs. Once that infrastructure is allocated though we can
enable this and let it fly!
Notable modifications from the current test suite today:
* Android tests are run in rustbuild instead of the makefiles, for whatever
reason I couldn't get the makefiles to work on Travis.
* A debuginfo test was updated to work with the current version of the Android
NDK.
* Some dependencies in `mk/tests.mk` were fixed to allow running tests in
parallel.
