This commit adds support to the build system to execute test suites that cannot
run natively but can instead run inside of a QEMU emulator. A proof-of-concept
builder was added for the `arm-unknown-linux-gnueabihf` target to show off how
this might work.
In general the architecture is to have a server running inside of the emulator
which a local client connects to. The protocol between the server/client
supports compiling tests on the host and running them on the target inside the
emulator.
Closes#33114
This commit adds a new tool, `build-manifest`, which is used to generate a
distribution manifest of all produced artifacts. This tool is intended to
replace the `build-rust-manifest.py` script that's currently located on the
buildmaster. The intention is that we'll have a builder which periodically:
* Downloads all artifacts for a commit
* Runs `./x.py dist hash-and-sign`. This will generate `sha256` and `asc` files
as well as TOML manifests.
* Upload all generated hashes and manifests to the directory the artifacts came
from.
* Upload *all* artifacts (tarballs and hashes and manifests) to an archived
location.
* If necessary, upload all artifacts to the main location.
This script is intended to just be the second step here where orchestrating
uploads and such will all happen externally from the build system itself.
This commit adds a new flag to the configure script,
`--enable-extended`, which is intended for specifying a desire to
compile the full suite of Rust tools such as Cargo, the RLS, etc. This
is also an indication that the build system should create combined
installers such as the pkg/exe/msi artifacts.
Currently the `--enable-extended` flag just indicates that combined
installers should be built, and Cargo is itself not compiled just yet
but rather only downloaded from its location. The intention here is to
quickly get to feature parity with the current release process and then
we can start improving it afterwards.
All new files in this PR inside `src/etc/installer` are copied from the
rust-packaging repository.
Fix rustbuild to work with --libdir.
Similar to the makefiles, pass CFG_LIBDIR_RELATIVE to cargo when building
rustc in stages > 0. This tells rustc to check the different directory.
I'm not sure how you want this handled in the toml system (my distribution, Gentoo, uses configure still). I have a feeling the system needs a rework anyways for rustbuild. If there is some discussion that needs to happen, could you merge this in the mean time? I'd be happy to help transition this to a better method.
Make rustbuild force_alloc_system rather than relying on stage0
This 'fixes' jemalloc-less local rebuilds, where we tell cargo that we're actually stage1 (this only fixes the rustbuild path, since I wasn't enthusiastic to dive into the makefiles).
There should be one effect from this PR: `--enable-local-rebuild --disable-jemalloc` will successfully build a stage0 std (rather than erroring). Ideally I think it'd be nice to specify an allocator preference in Cargo.toml/cargo command line (used when an allocator must be picked i.e. dylibs, not rlibs), but since that's not possible we can make do with a force_alloc_system feature. Sadly this locks you into a single allocator in the build libstd, making any eventual implementation of #38575 not quite right in this edge case, but clearly not many people exercise the combination of these two flags.
This PR is also a substitute for #37975 I think. The crucial difference is that the feature name here is distinct from the jemalloc feature (reused in the previous PR) - we don't want someone to be forced into alloc_system just for disabling jemalloc!
Fixes#39054
r? @alexcrichton
travis: Get an emscripten builder online
This commit adds a new entry to the Travis matrix which will execute emscripten
test suites. Along the way it updates a few bits of the test suite to continue
passing on emscripten, such as:
* Ignoring i128/u128 tests as they're presumably just not working (didn't
investigate as to why)
* Disabling a few process tests (not working on emscripten)
* Ignore some num tests in libstd (#39119)
* Fix some warnings when compiling
This commit adds a new entry to the Travis matrix which will execute emscripten
test suites. Along the way it updates a few bits of the test suite to continue
passing on emscripten, such as:
* Ignoring i128/u128 tests as they're presumably just not working (didn't
investigate as to why)
* Disabling a few process tests (not working on emscripten)
* Ignore some num tests in libstd (#39119)
* Fix some warnings when compiling
This expands the `cross` travis matrix entry with a few more targets that our
nightlies are building:
* x86_64-rumprun-netbsd
* arm-unknown-linux-musleabi
* arm-unknown-linux-musleabihf
* armv7-unknown-linux-musleabihf
* mips-unknown-linux-musl
* mipsel-unknown-linux-musl
This commit doesn't compile custom toolchains like our current cross-image does,
but instead compiles musl manually and then compiles libunwind manually (like
x86_64) for use for the ARM targets and just uses openwrt toolchains for the
mips targets.
rustbuild: Skip the build_helper crate in tests
I've been noticing some spurious recompiles of the final stage on Travis lately
and in debugging them I found a case where we were a little to eager to update
a stamp file due to the build_helper library being introduced during the testing
phase.
Part of the rustbuild system detects when libstd is recompiled and automatically
cleans out future directories to ensure that dirtyness propagation works. To do
this rustbuild doesn't know the artifact name of the standard library so it just
probes everything in the target directory, looking to see if anything changed.
The problem here happened where:
* First, rustbuild would compile everything (a normal build)
* Next, rustbuild would run all tests
* During testing, the libbuild_helper library was introduced into the target
directory, making it look like a change happened because a file is newer
than the newest was before
* Detecting a change, the next compilation would then cause rustbuild to clean
out old artifacts and recompile everything again.
This commit fixes this problem by correcting rustbuild to just not test the
build_helper crate at all. This crate doesn't have any unit tests, nor is it
intended to. That way the target directories should stay the same throughout
testing after a previous build.
rustbuild: Actually don't build stage0 target rustc
This was attempted in #38853 but erroneously forgot one more case of where the
compiler was compiled. This commit fixes that up and adds a test to ensure this
doesn't sneak back in.
I've been noticing some spurious recompiles of the final stage on Travis lately
and in debugging them I found a case where we were a little to eager to update
a stamp file due to the build_helper library being introduced during the testing
phase.
Part of the rustbuild system detects when libstd is recompiled and automatically
cleans out future directories to ensure that dirtyness propagation works. To do
this rustbuild doesn't know the artifact name of the standard library so it just
probes everything in the target directory, looking to see if anything changed.
The problem here happened where:
* First, rustbuild would compile everything (a normal build)
* Next, rustbuild would run all tests
* During testing, the libbuild_helper library was introduced into the target
directory, making it look like a change happened because a file is newer
than the newest was before
* Detecting a change, the next compilation would then cause rustbuild to clean
out old artifacts and recompile everything again.
This commit fixes this problem by correcting rustbuild to just not test the
build_helper crate at all. This crate doesn't have any unit tests, nor is it
intended to. That way the target directories should stay the same throughout
testing after a previous build.
This was attempted in #38853 but erroneously forgot one more case of where the
compiler was compiled. This commit fixes that up and adds a test to ensure this
doesn't sneak back in.
This commit starts adding the infrastructure for uploading release artifacts
from AppVeyor/Travis on each commit. The idea is that eventually we'll upload a
full release to AppVeyor/Travis in accordance with plans [outlined earlier].
Right now this configures Travis/Appveyor to upload all tarballs in the `dist`
directory, and various images are updated to actually produce tarballs in these
directories. These are nowhere near ready to be actual release artifacts, but
this should allow us to play around with it and test it out. Once this commit
lands we should start seeing artifacts uploaded on each commit.
[outlined earlier]: https://internals.rust-lang.org/t/rust-ci-release-infrastructure-changes/4489
rustbuild: Implement DESTDIR support
This commit primarily starts supporting the `DESTDIR` environment variable like
the old build system. Along the way this brings `config.toml` up to date with
support in `config.mk` with install options supported.
Closes#38441
In #37280 we enabled line number debugging information in release artifacts,
primarily to close out #36452 where debugging information was critical for MSVC
builds of Rust to be useful in production. This commit, however, apparently had
some unfortunate side effects.
Namely it was noticed in #37477 that if `RUST_BACKTRACE=1` was set then any
compiler error would take a very long time for the compiler to exit. The cause
of the problem here was somewhat deep:
* For all compiler errors, the compiler will `panic!` with a known value. This
tears down the main compiler thread and allows cleaning up all the various
resources. By default, however, this panic output is suppressed for "normal"
compiler errors.
* When `RUST_BACKTRACE=1` was set this caused every compiler error to generate a
backtrace.
* The libbacktrace library hits a pathological case where it spends a very long
time in its custom allocation function, `backtrace_alloc`, because the
compiler has so much debugging information. More information about this can be
found in #29293 with a summary at the end of #37477.
To solve this problem this commit simply removes debuginfo from the compiler but
not from the standard library. This should allow us to keep #36452 closed while
also closing #37477. I've measured the difference to be orders of magnitude
faster than it was before, so we should see a much quicker time-to-exit after a
compile error when `RUST_BACKTRACE=1` is set.
Closes#37477Closes#37571
Don't restrict docs in compiler-docs mode
Search is broken without this. We want all crates to be included in compiler-docs mode. This was changed in https://github.com/rust-lang/rust/pull/38858, this PR brings that functionality back in compiler-docs mode.
rustbuild: Don't build target compilers in stage0
The `doc-book` and `doc-nomicon` steps accidentally depended on a rustbook
compiled by a cross-compiled compiler, which isn't necessary. Be sure to set the
`host` on these dependency edges to the build compiler to ensure that we're
always using a tool compiled for the host platform.
This was discovered trawling the build logs for the new dist bots and
discovering that they're building one too many compilers in stage0.
The `doc-book` and `doc-nomicon` steps accidentally depended on a rustbook
compiled by a cross-compiled compiler, which isn't necessary. Be sure to set the
`host` on these dependency edges to the build compiler to ensure that we're
always using a tool compiled for the host platform.
This was discovered trawling the build logs for the new dist bots and
discovering that they're building one too many compilers in stage0.
rustbuild: Quickly `dist` cross-host compilers
This commit optimizes the compile time for creating tarballs of cross-host
compilers and as a proof of concept adds two to the standard Travis matrix. Much
of this commit is further refactoring and refining of the `step.rs` definitions
along with the interpretation of `--target` and `--host` flags. This has gotten
confusing enough that I've also added a small test suite to
`src/bootstrap/step.rs` to ensure what we're doing works and doesn't regress.
After this commit when you execute:
./x.py dist --host $MY_HOST --target $MY_HOST
the build system will compile two compilers. The first is for the build platform
and the second is for the host platform. This second compiler is then packaged
up and placed into `build/dist` and is ready to go. With a fully cached LLVM and
docker image I was able to create a cross-host compiler in around 20 minutes
locally.
Eventually we plan to add a whole litany of cross-host entries to the Travis
matrix, but for now we're just adding a few before we eat up all the extra
capacity.
cc #38531
This commit optimizes the compile time for creating tarballs of cross-host
compilers and as a proof of concept adds two to the standard Travis matrix. Much
of this commit is further refactoring and refining of the `step.rs` definitions
along with the interpretation of `--target` and `--host` flags. This has gotten
confusing enough that I've also added a small test suite to
`src/bootstrap/step.rs` to ensure what we're doing works and doesn't regress.
After this commit when you execute:
./x.py dist --host $MY_HOST --target $MY_HOST
the build system will compile two compilers. The first is for the build platform
and the second is for the host platform. This second compiler is then packaged
up and placed into `build/dist` and is ready to go. With a fully cached LLVM and
docker image I was able to create a cross-host compiler in around 20 minutes
locally.
Eventually we plan to add a whole litany of cross-host entries to the Travis
matrix, but for now we're just adding a few before we eat up all the extra
capacity.
cc #38531
Recent versions of Cargo lift less output up into the "main" directory, so let's
look more inside the `deps` folder for changes to propagate differences.
Closes#38744Closes#38746
Despite what the comment says, we actually need to do this. We're not cleaning
out the stage0 compiler's sysroot, but rather just our own sysroot that we
assembled previously.
Gate on distcheck on Travis
This commit adds a new entry to the Travis matrix to gate on distcheck, the illustrious test process that has historically taken *8 hours* to complete and also breaks all the time on nightly. By adding it to Travis we should hope to never see nightly breakage (like https://github.com/rust-lang/rust/issues/38690) because of this ever again!
"But wait, surely we can't wait 8 hours for all PRs!" you might be thinking, and you are indeed correct. The distcheck added here is much more optimized for speed than the old buildbot instances for a number of reasons:
* We're not building *two host compilers* beforehand. The current distcheck bot does a cross for i686 Linux and x86_64 Linux before it actually runs distcheck, building 6 compilers and LLVM twice. None of this is done in parallel as well (e.g. `-j1`). Not doing any of this work will be a huge win!
* We're using sccache to compile LLVM, so it should be much faster. Distcheck on the bots didn't cache LLVM well and rebuilt it every time.
All in all, this version of "distcheck" should be exactly like other matrix entries that run tests except that it's a *little* slower to start as it has to create the source tarball then rebuild the build system in the distcheck dir. Overall this should be well under the 2 hours that Android is currently taking anyway.
Closes https://github.com/rust-lang/rust/issues/38691
rustbuild: Compile all support tools in stage0
This commit changes all tools and such to get compiled in stage0, not in
later stages. The purpose of this commit is to cut down dependencies on later
stages for future modifications to the build system. Notably we're going to be
adding builders that produce a full suite of cross-compiled artifacts for a
particular host, and that shouldn't compile the `x86_64-unknown-linux-gnu`
compiler more than once. Currently dependencies on, for example, the error index
end up compiling the `x86_64-unknown-linux-gnu` compiler more than necessary.
As a result here we move many dependencies on these tools to being produced by a
stage0 compiler, not a stage1+ compiler. None of these tools actually need to be
staged at all, so they'll exhibit consistent behavior across the stages.
The android-copy-libs step is crucial for running tests on the Android target as
it copies necessary scripts and such to the emulator. We must run that before
running any tests there, but we erroneously only did it for compiletest test
suites!
