Currently our slowest test suite on android, run-pass, takes over 5 times longer
than the x86_64 component (~400 -> ~2200s). Typically QEMU emulation does indeed
add overhead, but not 5x for this kind of workload. One of the slowest parts of
the Android process is that *compilation* happens serially. Tests themselves
need to run single-threaded on the emulator (due to how the test harness works)
and this forces the compiles themselves to be single threaded.
Now Travis gives us more than one core per machine, so it'd be much better if we
could take advantage of them! The emulator itself is still fundamentally
single-threaded, but we should see a nice speedup by sending binaries for it to
run much more quickly.
It turns out that we've already got all the tools to do this in-tree. The
qemu-test-{server,client} that are in use for the ARM Linux testing are a
perfect match for the Android emulator. This commit migrates the custom adb
management code in compiletest/rustbuild to the same qemu-test-{server,client}
implementation that ARM Linux uses.
This allows us to lift the parallelism restriction on the compiletest test
suites, namely run-pass. Consequently although we'll still basically run the
tests themselves in single threaded mode we'll be able to compile all of them in
parallel, keeping the pipeline much more full and using more cores for the work
at hand. Additionally the architecture here should be a bit speedier as it
should have less overhead than adb which is a whole new process on both the host
and the emulator!
Locally on an 8 core machine I've seen the run-pass test suite speed up from
taking nearly an hour to only taking 6 minutes. I don't think we'll see quite a
drastic speedup on Travis but I'm hoping this change can place the Android tests
well below 2 hours instead of just above 2 hours.
Because the client/server here are now repurposed for more than just QEMU,
they've been renamed to `remote-test-{server,client}`.
Note that this PR does not currently modify how debuginfo tests are executed on
Android. While parallelizable it wouldn't be quite as easy, so that's left to
another day. Thankfully that test suite is much smaller than the run-pass test
suite.
As a final fix I discovered that the ARM and Android test suites were actually
running all library unit tests (e.g. stdtest, coretest, etc) twice. I've
corrected that to only run tests once which should also give a nice boost in
overall cycle time here.
Now that the final bug fixes have been merged into sccache we can start
leveraging sccache on the MSVC builders on AppVeyor instead of relying on the
ad-hoc caching strategy of trigger files and whatnot.
Shrink the rust-src component
Before this change, the installable rust-src component had essentially the same contents as the rustc-src dist tarball, just additionally wrapped in a rust-installer. As discussed on [internals], rust-src is only meant to support uses for the standard library, so it doesn't really need the rest of the compiler sources.
Now rust-src only contains libstd and its path dependencies, which roughly matches the set of crates that have rust-analysis data. The result is **significantly** smaller, from 36MB to 1.3MB compressed, and from 247MB to 8.5MB uncompressed.
[internals]: https://internals.rust-lang.org/t/minimizing-the-rust-src-component/5117
Add Hexagon support
This requires an updated LLVM with https://reviews.llvm.org/D31999 and https://reviews.llvm.org/D32000 to build libcore.
A basic hello world builds and runs successfully on the hexagon simulator. libcore is fine with LLVM fixes, but libstd requires a lot more work since there's a custom rtos running on most hexagon cores. Running Linux sounds possible though, so maybe getting linux + musl going would be easier.
Here's the target file I've been using for testing
```
{
"arch": "hexagon",
"llvm-target": "hexagon-unknown-elf",
"os": "none",
"target-endian": "little",
"target-pointer-width": "32",
"data-layout": "e-m:e-p:32:32:32-a:0-n16:32-i64:64:64-i32:32:32-i16:16:16-i1:8:8-f32:32:32-f64:64:64-v32:32:32-v64:64:64-v512:512:512-v1024:1024:1024-v2048:2048:2048",
"linker": "hexagon-clang",
"linker-flavor": "gcc",
"executables": true,
"cpu": "hexagonv60"
}
```
Support AddressSanitizer and ThreadSanitizer on x86_64-apple-darwin
[ASan](https://clang.llvm.org/docs/AddressSanitizer.html#supported-platforms) and [TSan](https://clang.llvm.org/docs/ThreadSanitizer.html#supported-platforms) are supported on macOS, and this commit enables their support.
The sanitizers are always built as `*.dylib` on Apple platforms, so they cannot be statically linked into the corresponding `rustc_?san.rlib`. The dylibs are directly copied to `lib/rustlib/x86_64-apple-darwin/lib/` instead.
Note, although Xcode also ships with their own copies of ASan/TSan dylibs, we cannot use them due to version mismatch.
----
~~There is a caveat: the sanitizer libraries are linked as `@rpath/` (due to https://reviews.llvm.org/D6018), so the user needs to additionally pass `-C rpath`:~~
**Edit:** Passing rpath is now automatic.
Adds rust-windbg.cmd script
Adds rust-gdb/rust-lldb equivalent for windbg that loads the Rust .natvis files on start.
This change modifies the bootstrap code to add rust-windbg to bin and the .natvis files to lib/rustlib/etc.
Example usage from cmd or PowerShell:
```
rust-windbg -c "bu rs_f442289d74765418!rs::main;g" target\debug\rs.exe
```
Run tests for the cargo submodule in tree
Previously the `cargotest` suite would run some arbitrary revision of Cargo's
test suite, but now that we're bundling it in tree we should be running the
Cargo submodule's test suite instead.
ASan and TSan are supported on macOS, and this commit enables their
support.
The sanitizers are always built as *.dylib on Apple platforms, so they
cannot be statically linked into the corresponding `rustc_?san.rlib`. The
dylibs are directly copied to `lib/rustlib/x86_64-apple-darwin/lib/`
instead.
Note, although Xcode also ships with their own copies of ASan/TSan dylibs,
we cannot use them due to version mismatch.
There is a caveat: the sanitizer libraries are linked as @rpath, so the
user needs to additionally pass `-C rpath`:
rustc -Z sanitizer=address -C rpath file.rs
^~~~~~~~
Otherwise there will be a runtime error:
dyld: Library not loaded: @rpath/libclang_rt.asan_osx_dynamic.dylib
Referenced from: /path/to/executable
Reason: image not found
Abort trap: 6
The next commit includes a temporary change in compiler to force the linker
to emit a usable @rpath.
LLVM 4.0 Upgrade
Since nobody has done this yet, I decided to get things started:
**Todo:**
* [x] push the relevant commits to `rust-lang/llvm` and `rust-lang/compiler-rt`
* [x] cleanup `.gitmodules`
* [x] Verify if there are any other commits from `rust-lang/llvm` which need backporting
* [x] Investigate / fix debuginfo ("`<optimized out>`") failures
* [x] Use correct emscripten version in docker image
---
Closes#37609.
---
**Test results:**
Everything is green 🎉
Previously the `cargotest` suite would run some arbitrary revision of Cargo's
test suite, but now that we're bundling it in tree we should be running the
Cargo submodule's test suite instead.
Disable debuginfo when compiling tools
Currently the Cargo binary has jumped from 14M to 34M on the beta channel, which
appears to be due to the fact that we're compiling tools with debug information
inside them. This additionally means that the `rls` binary is 62M right now!
This wasn't an intentional change, so be sure to disable debuginfo when
compiling tools as it's just intended for the standard library and compile for
now.
Use the existing path when removing the prefix fails
This allows the use of out-of-tree paths to be specified. I found this while trying to build with a modified version of `rls-data`, which is currently pointing to a version on crates.io.
cc @alexcrichton
Also, it wasn't clear if I needed to add a test for this (or how). I didn't see any tests that took paths into consideration.
Currently the Cargo binary has jumped from 14M to 34M on the beta channel, which
appears to be due to the fact that we're compiling tools with debug information
inside them. This additionally means that the `rls` binary is 62M right now!
This wasn't an intentional change, so be sure to disable debuginfo when
compiling tools as it's just intended for the standard library and compile for
now.
This commit knocks out a longstanding FIXME in rustbuild which should correctly
recompile stage0 compiletest and such whenever libstd itself changes. The
solution implemented here was to implement a notion of "order only" dependencies
and then add a new dependency stage for clearing out the tools dir, using
order-only deps to ensure that it happens correctly.
The dependency drawing for tools is a bit wonky now but I think this'll get the
job done.
Closes#39396
This commit enables the `rust-analysis` package to be produced for all targets
that are part of the `dist-*` suite of docker images on Travis. Currently
these packages are showing up with `available = false` in the
`channel-rust-nightly.toml` manifest where we'd prefer to have them show up for
all targets.
Unfortunately rustup isn't handling the `available = false` section well right
now, so this should also inadvertently fix the nightly regression.
Add the RLS as a submodule and build a package out of it
r? @brson (and cc @alexcrichton) Please review closely, I am not at all convinced I've done the right things here. I did run `x.py dist` and it makes an rls package which looks right to my eyes, but I haven't tested on non-linux platforms nor am I really sure what it should look like.
This does not attempt to run tests for the RLS yet.
* Use the right version when building combined installer
* Update dependencies of rls as it depends on rustc and plugins
* Fix build-manifest and the versions it uses for the rls