Build instruction profiler runtime as part of compiler-rt
r? @alexcrichton
This is #38608 with some fixes.
Still missing:
- [x] testing with profiler enabled on some builders (on which ones? Should I add the option to some of the already existing configurations, or create a new configuration?);
- [x] enabling distribution (on which builders?);
- [x] documentation.
The rationale is that BOM stripping is needed for lazy source loading
for external crates, and duplication can be avoided by moving the
corresponding functionality to libsyntax_pos.
Update rust-installer for Windows executable mode
It now marks a few whitelisted extensions as executable in the tarball,
so Windows packages can be extracted on other platforms and directly
execute install.sh.
It also includes a fix for the chmod on bulk dirs, so now the html docs
won't be marked executable en masse.
Fixes#42121
r? @alexcrichton
It now marks a few whitelisted extensions as executable in the tarball,
so Windows packages can be extracted on other platforms and directly
execute install.sh.
It also includes a fix for the chmod on bulk dirs, so now the html docs
won't be marked executable en masse.
Fixes#42121
r? @alexcrichton
This avoids double compiled Cargo. Hopefully this would speed up (extended) compilation for ~10m.
Notes: when updating Cargo submodule, the replacement version may also need to be updated.
Make only rustc_trans depend on rustc_llvm
With these changes, only rustc_trans depends directly on rustc_llvm (and no crate gained a new dependency on trans). This means changing LLVM doesn't rebuild librustc or rustc_metadata, only rustc_trans, rustc_driver and the rustc executable
Also, rustc_driver technically doesn't know about LLVM any more (of course, it still handles a ton of options that conceptually refer to LLVM, but it delegates their implementation to trans).
What I *didn't* implement was merging most or all of rustc_llvm into rustc_trans. I ran into a nasty bug, which was probably just a silly typo somewhere but I probably won't have the time to figure it out in the next week or two. I opened #41699 for that step.
Fixes#41473
This does not actually improve build times, since it still depends
on rustc_trans, but is better layering and fits the multi-backend
future slightly better.
Consequently, session creation can no longer initialize LLVM.
The few places that use the compiler without going through
rustc_driver/CompilerCalls thus need to be careful to manually
initialize LLVM (via rustc_trans!) immediately after session
creation.
This means librustc is not rebuilt when LLVM changes.
Move the code for loading metadata from rlibs and dylibs from
rustc_metadata into rustc_trans, and introduce a trait to avoid
introducing a direct dependency on rustc_trans.
This means rustc_metadata is no longer rebuilt when LLVM changes.
When -Z profile is passed, the GCDAProfiling LLVM pass is added
to the pipeline, which uses debug information to instrument the IR.
After compiling with -Z profile, the $(OUT_DIR)/$(CRATE_NAME).gcno
file is created, containing initial profiling information.
After running the program built, the $(OUT_DIR)/$(CRATE_NAME).gcda
file is created, containing branch counters.
The created *.gcno and *.gcda files can be processed using
the "llvm-cov gcov" and "lcov" tools. The profiling data LLVM
generates does not faithfully follow the GCC's format for *.gcno
and *.gcda files, and so it will probably not work with other tools
(such as gcov itself) that consume these files.
use diff crate for compile-fail test diagnostics #41474
Hello!
This fixes#41474
We were using a custom implementation to dump the differences between expected and actual outputs of compile-fail tests.
I removed this internal implementation and added `diff` crate as a new dependency to `compile-fail`.
Again, huge thanks to @nikomatsakis for guiding.
Currently our slowest test suite on android, run-pass, takes over 5 times longer
than the x86_64 component (~400 -> ~2200s). Typically QEMU emulation does indeed
add overhead, but not 5x for this kind of workload. One of the slowest parts of
the Android process is that *compilation* happens serially. Tests themselves
need to run single-threaded on the emulator (due to how the test harness works)
and this forces the compiles themselves to be single threaded.
Now Travis gives us more than one core per machine, so it'd be much better if we
could take advantage of them! The emulator itself is still fundamentally
single-threaded, but we should see a nice speedup by sending binaries for it to
run much more quickly.
It turns out that we've already got all the tools to do this in-tree. The
qemu-test-{server,client} that are in use for the ARM Linux testing are a
perfect match for the Android emulator. This commit migrates the custom adb
management code in compiletest/rustbuild to the same qemu-test-{server,client}
implementation that ARM Linux uses.
This allows us to lift the parallelism restriction on the compiletest test
suites, namely run-pass. Consequently although we'll still basically run the
tests themselves in single threaded mode we'll be able to compile all of them in
parallel, keeping the pipeline much more full and using more cores for the work
at hand. Additionally the architecture here should be a bit speedier as it
should have less overhead than adb which is a whole new process on both the host
and the emulator!
Locally on an 8 core machine I've seen the run-pass test suite speed up from
taking nearly an hour to only taking 6 minutes. I don't think we'll see quite a
drastic speedup on Travis but I'm hoping this change can place the Android tests
well below 2 hours instead of just above 2 hours.
Because the client/server here are now repurposed for more than just QEMU,
they've been renamed to `remote-test-{server,client}`.
Note that this PR does not currently modify how debuginfo tests are executed on
Android. While parallelizable it wouldn't be quite as easy, so that's left to
another day. Thankfully that test suite is much smaller than the run-pass test
suite.
As a final fix I discovered that the ARM and Android test suites were actually
running all library unit tests (e.g. stdtest, coretest, etc) twice. I've
corrected that to only run tests once which should also give a nice boost in
overall cycle time here.
Prior to this commit, the contents of the Unstable Book were assumed to
be unstable features. This commit moves features into 'language features'
or 'library features' subsections. It also moves the 'linker_flavor'
compiler flag into a new 'Compiler Flags' subsection.
Even though it was helpful, I removed the tidy check that
cross-references the SUMMARY.md links with the Unstable Book directory
contents just because it would be difficult to maintain.
Relevant PR: https://github.com/rust-lang/rust/issues/41142.
Add a tidy lint that checks for...
* Unstable Book sections with no corresponding SUMMARY.md links
* unstable features that don't have Unstable Book sections
* Unstable Book sections that don't have corresponding unstable features
This commit deletes the internal liblog in favor of the implementation that
lives on crates.io. Similarly it's also setting a convention for adding crates
to the compiler. The main restriction right now is that we want compiler
implementation details to be unreachable from normal Rust code (e.g. requires a
feature), and by default everything in the sysroot is reachable via `extern
crate`.
The proposal here is to require that crates pulled in have these lines in their
`src/lib.rs`:
#![cfg_attr(rustbuild, feature(staged_api, rustc_private))]
#![cfg_attr(rustbuild, unstable(feature = "rustc_private", issue = "27812"))]
This'll mean that by default they're not using these attributes but when
compiled as part of the compiler they do a few things:
* Mark themselves as entirely unstable via the `staged_api` feature and the
`#![unstable]` attribute.
* Allow usage of other unstable crates via `feature(rustc_private)` which is
required if the crate relies on any other crates to compile (other than std).
This causes problems when first cloning and bootstrapping the repository
unfortunately, so let's ensure that Cargo sticks around in its own workspace.
Because Cargo is a submodule it's not available by default on the inital clone
of the rust-lang/rust repository. Normally it's the responsibility of the
rustbuild to take care of this, but unfortunately to build rustbuild itself we
need to resolve the workspace conflicts.
To deal with this we'll just have to ensure that all submodules are in their own
workspace, which sort of makes sense anyway as updates to dependencies as
bugfixes to Cargo should go to rust-lang/cargo instead of rust-lang/rust. In any
case this commit removes Cargo from the global workspace which should resolve
the issues that we've been seeing.
To actually perform this the `cargo` submodule has been moved to the top
directory to ensure it's outside the scope of `src/Cargo.toml` as a workspace.
This commit adds support to rustbuild for compiling Cargo as part of the release
process. Previously rustbuild would simply download a Cargo snapshot and
repackage it. With this change we should be able to turn off artifacts from the
rust-lang/cargo repository and purely rely on the artifacts Cargo produces here.
The infrastructure added here is intended to be extensible to other components,
such as the RLS. It won't exactly be a one-line addition, but the addition of
Cargo didn't require too much hooplah anyway.
The process for release Cargo will now look like:
* The rust-lang/rust repository has a Cargo submodule which is used to build a
Cargo to pair with the rust-lang/rust release
* Periodically we'll update the cargo submodule as necessary on rust-lang/rust's
master branch
* When branching beta we'll create a new branch of Cargo (as we do today), and
the first commit to the beta branch will be to update the Cargo submodule to
this exact revision.
* When branching stable, we'll ensure that the Cargo submodule is updated and
then make a stable release.
Backports to Cargo will look like:
* Send a PR to cargo's master branch
* Send a PR to cargo's release branch (e.g. rust-1.16.0)
* Send a PR to rust-lang/rust's beta branch updating the submodule
* Eventually send a PR to rust-lang/rust's master branch updating the submodule
For reference, the process to add a new component to the rust-lang/rust release
would look like:
* Add `$foo` as a submodule in `src/tools`
* Add a `tool-$foo` step which compiles `$foo` with the specified compiler,
likely mirroring what Cargo does.
* Add a `dist-$foo` step which uses `src/tools/$foo` and the `tool-$foo` output
to create a rust-installer package for `$foo` likely mirroring what Cargo
does.
* Update the `dist-extended` step with a new dependency on `dist-$foo`
* Update `src/tools/build-manifest` for the new component.
test: Verify all sysroot crates are unstable
As we continue to add more crates to the compiler and use them to implement
various features we want to be sure we're not accidentally expanding the API
surface area of the compiler! To that end this commit adds a new `run-make` test
which will attempt to `extern crate foo` all crates in the sysroot, verifying
that they're all unstable.
This commit discovered that the `std_shim` and `test_shim` crates were
accidentally stable and fixes the situation by deleting those shims. The shims
are no longer necessary due to changes in Cargo that have happened since they
were originally incepted.
As we continue to add more crates to the compiler and use them to implement
various features we want to be sure we're not accidentally expanding the API
surface area of the compiler! To that end this commit adds a new `run-make` test
which will attempt to `extern crate foo` all crates in the sysroot, verifying
that they're all unstable.
This commit discovered that the `std_shim` and `test_shim` crates were
accidentally stable and fixes the situation by deleting those shims. The shims
are no longer necessary due to changes in Cargo that have happened since they
were originally incepted.
mdbook has a lot of optional dependencies that we don't want, so instead
of using it directly, we re-build rustbook to use mdbook as a library.
For convenience' sake, we keep the same CLI interface as mdbook; the
only difference is that it only accepts build and test subcommands,
rather than the full range.
It claims to accept most GNU linker options, but in fact most of them
have no effect and instead it requires some special options which are
easier to handle in a separate trait.
Currently added:
- `export_symbols`: works on executables as special Emscripten case
since staticlibs/dylibs aren't compiled to JS, while exports are
required to be accessible from JS.
Fixes#39171.
- `optimize` - translates Rust's optimization level to Emscripten
optimization level (whether passed via `-C opt-level=...` or `-O...`).
Fixes#36899.
- `debuginfo` - translates debug info; Emscripten has 5 debug levels
while Rust has 3, so chose to translate `-C debuginfo=1` to `-g3`
(preserves whitespace, variable and function names for easy debugging).
Fixes#36901.
- `no_default_libraries` - tells Emscripten to exlude `memcpy` and co.
Delete the makefile build system
This PR deletes the makefile build system in favor of the rustbuild build system. The beta has now been branched so 1.16 will continue to be buildable from the makefiles, but going forward 1.17 will only be buildable with rustbuild.
Rustbuild has been the default build system [since 1.15.0](https://github.com/rust-lang/rust/pull/37817) and the makefiles were [proposed for deletion](https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368) at this time back in November of last year.
And now with the deletion of these makefiles we can start getting those sweet sweet improvements of using crates.io crates in the compiler!
Add support for test suites emulated in QEMU
This commit adds support to the build system to execute test suites that cannot
run natively but can instead run inside of a QEMU emulator. A proof-of-concept
builder was added for the `arm-unknown-linux-gnueabihf` target to show off how
this might work.
In general the architecture is to have a server running inside of the emulator
which a local client connects to. The protocol between the server/client
supports compiling tests on the host and running them on the target inside the
emulator.
Closes#33114
Miscellaneous refactors around how lints and typeck interact
This is preparation for making incr. comp. skip typeck. The main gist of is trying to rationalize the outputs from typeck that are not part of tables:
- one bit of output is the `used_trait_imports` set, which becomes something we track for dependencies
- the other big of output are various lints; we used to store these into a table on sess, but this work stores them into the`TypeckTables`, and then makes the lint pass consult that
- I think it probably makes sense to handle errors similarly, eventually, but that's not necessary now
r? @eddyb
Fixes#39495
Bump version, upgrade bootstrap
This commit updates the version number to 1.17.0 as we're not on that version of
the nightly compiler, and at the same time this updates src/stage0.txt to
bootstrap from freshly minted beta compiler and beta Cargo.
This commit updates the version number to 1.17.0 as we're not on that version of
the nightly compiler, and at the same time this updates src/stage0.txt to
bootstrap from freshly minted beta compiler and beta Cargo.
This commit adds support to the build system to execute test suites that cannot
run natively but can instead run inside of a QEMU emulator. A proof-of-concept
builder was added for the `arm-unknown-linux-gnueabihf` target to show off how
this might work.
In general the architecture is to have a server running inside of the emulator
which a local client connects to. The protocol between the server/client
supports compiling tests on the host and running them on the target inside the
emulator.
Closes#33114
This commit adds a new tool, `build-manifest`, which is used to generate a
distribution manifest of all produced artifacts. This tool is intended to
replace the `build-rust-manifest.py` script that's currently located on the
buildmaster. The intention is that we'll have a builder which periodically:
* Downloads all artifacts for a commit
* Runs `./x.py dist hash-and-sign`. This will generate `sha256` and `asc` files
as well as TOML manifests.
* Upload all generated hashes and manifests to the directory the artifacts came
from.
* Upload *all* artifacts (tarballs and hashes and manifests) to an archived
location.
* If necessary, upload all artifacts to the main location.
This script is intended to just be the second step here where orchestrating
uploads and such will all happen externally from the build system itself.
i128 and u128 support
Brings i128 and u128 support to nightly rust, behind a feature flag. The goal of this PR is to do the bulk of the work for 128 bit integer support. Smaller but just as tricky features needed for stabilisation like 128 bit enum discriminants are left for future PRs.
Rebased version of #37900, which in turn was a rebase + improvement of #35954 . Sadly I couldn't reopen#37900 due to github. There goes my premium position in the homu queue...
[plugin-breaking-change]
cc #35118 (tracking issue)
This commit changes all tools and such to get compiled in stage0, not in
later stages. The purpose of this commit is to cut down dependencies on later
stages for future modifications to the build system. Notably we're going to be
adding builders that produce a full suite of cross-compiled artifacts for a
particular host, and that shouldn't compile the `x86_64-unknown-linux-gnu`
compiler more than once. Currently dependencies on, for example, the error index
end up compiling the `x86_64-unknown-linux-gnu` compiler more than necessary.
As a result here we move many dependencies on these tools to being produced by a
stage0 compiler, not a stage1+ compiler. None of these tools actually need to be
staged at all, so they'll exhibit consistent behavior across the stages.
A tweak was made to dependencies in #38451 but the makefiles weren't updated to
accompany this. Instead of trying to integerate the `build_helper` crate into
the makefiles (which currently isn't present) this commit takes the approach of
just duplicating the required logic, which should be small enough for now.
This commit updates the gcc-rs dependency to 0.3.40 to pick up a fix for i686
musl where we needed to pass an extra linker flag to get autoconf's detection of
executables working correctly.
In LLVM 4.0, this enum becomes an actual type-safe enum, which breaks
all of the interfaces. Introduce our own copy of the bitflags that we
can then safely convert to the LLVM one.
[LLVM 4.0] Use llvm::Attribute APIs instead of "raw value" APIs
The latter will be removed in LLVM 4.0 (see 4a6fc8bacf).
The librustc_llvm API remains mostly unchanged, except that llvm::Attribute is no longer a bitflag but represents only a *single* attribute.
The ability to store many attributes in a small number of bits and modify them without interacting with LLVM is only used in rustc_trans::abi and closely related modules, and only attributes for function arguments are considered there.
Thus rustc_trans::abi now has its own bit-packed representation of argument attributes, which are translated to rustc_llvm::Attribute when applying the attributes.
cc #37609
The librustc_llvm API remains mostly unchanged, except that llvm::Attribute is no longer a bitflag but represents only a *single* attribute.
The ability to store many attributes in a small number of bits and modify them without interacting with LLVM is only used in rustc_trans::abi and closely related modules, and only attributes for function arguments are considered there.
Thus rustc_trans::abi now has its own bit-packed representation of argument attributes, which are translated to rustc_llvm::Attribute when applying the attributes.
A few changes are included here:
* The `winapi` and `url` dependencies were dropped. The source code for these
projects is pretty weighty, and we're about to vendor them, so let's not
commit to that intake just yet. If necessary we can vendor them later but for
now it shouldn't be necessary.
* The `--frozen` flag is now always passed to Cargo, obviating the need for
tidy's `cargo_lock` check.
* Tidy was updated to not check the vendor directory
Closes#34687
This commit is a rewrite of the user-facing interface to the rustbuild build
system. The intention here is to make it much easier to compile/test the project
without having to remember weird rule names and such. An overall view of the new
interface is:
# build everything
./x.py build
# document everyting
./x.py doc
# test everything
./x.py test
# test libstd
./x.py test src/libstd
# build libcore stage0
./x.py build src/libcore --stage 0
# run stage1 run-pass tests
./x.py test src/test/run-pass --stage 1
The `src/bootstrap/bootstrap.py` script is now aliased as a top-level `x.py`
script. This `x` was chosen to be both short and easily tab-completable (no
collisions in that namespace!). The build system now accepts a "subcommand" of
what to do next, the main ones being build/doc/test.
Each subcommand then receives an optional list of arguments. These arguments are
paths in the source repo of what to work with. That is, if you want to test a
directory, you just pass that directory as an argument.
The purpose of this rewrite is to do away with all of the arcane renames like
"rpass" is the "run-pass" suite, "cfail" is the "compile-fail" suite, etc. By
simply working with directories and files it's much more intuitive of how to run
a test (just pass it as an argument).
The rustbuild step/dependency management was also rewritten along the way to
make this easy to work with and define, but that's largely just a refactoring of
what was there before.
The *intention* is that this support is extended for arbitrary files (e.g.
`src/test/run-pass/my-test-case.rs`), but that isn't quite implemented just yet.
Instead directories work for now but we can follow up with stricter path
filtering logic to plumb through all the arguments.
Split up libproc_macro_plugin
Separate the plugin code from non-plugin code to break a potential cycle in crates.
This will allow us to merge the new libproc_macro_tokens into libproc_macro.
r? @alexcrichton
Separate the plugin code from non-plugin code to break a potential cycle in crates.
This will allow us to merge the new libproc_macro_tokens into libproc_macro.
As the entry point for building the Rust compiler, a good user experience hinges
on this compiling quickly to get to the meat of the problem. To that end use
`#[cfg]`-specific dependencies to avoid building Windows crates on Unix and drop
the `regex` crate for now which was easily replacable with some string
searching.
This updates the commit to use workspaces to use `cargo metadata` instead of
hardcoded lists about what to test. This should help us be resilient to updates
in the future on behalf of the crate DAG and minimize the amount of files that
need to be touched.
This involves hacking the code used to run cargo test on various
packages, because it reads Cargo.lock to determine which packages should
be tested. This change implements a blacklist, since that will catch new
crates when they are added in the future.
