This commit is an implementation of [RFC 1183][rfc] which allows swapping out
the default allocator on nightly Rust. No new stable surface area should be
added as a part of this commit.
[rfc]: https://github.com/rust-lang/rfcs/pull/1183
Two new attributes have been added to the compiler:
* `#![needs_allocator]` - this is used by liballoc (and likely only liballoc) to
indicate that it requires an allocator crate to be in scope.
* `#![allocator]` - this is a indicator that the crate is an allocator which can
satisfy the `needs_allocator` attribute above.
The ABI of the allocator crate is defined to be a set of symbols that implement
the standard Rust allocation/deallocation functions. The symbols are not
currently checked for exhaustiveness or typechecked. There are also a number of
restrictions on these crates:
* An allocator crate cannot transitively depend on a crate that is flagged as
needing an allocator (e.g. allocator crates can't depend on liballoc).
* There can only be one explicitly linked allocator in a final image.
* If no allocator is explicitly requested one will be injected on behalf of the
compiler. Binaries and Rust dylibs will use jemalloc by default where
available and staticlibs/other dylibs will use the system allocator by
default.
Two allocators are provided by the distribution by default, `alloc_system` and
`alloc_jemalloc` which operate as advertised.
Closes#27389
New enough find on Linux doesn't support "-perm +..." and suggests
using "-perm /..." instead, but that doesn't work on Windows.
Hopefully all platforms are happy with this expanded version.
I don't have access to a Windows development system to test this, so someone needs to verify that this actually works there before merging.
Closes#19981.
Per @steveklabnik's comment [here](https://github.com/rust-lang/cargo/issues/739#issuecomment-130085860), the Pandoc components of the Makefile are no longer used, and as such the corresponding components of the documentation are out of date.
- I've removed the Pandoc (and therefore also LaTeX) elements of the makefile and confirmed that the build proceeds correctly.
- I updated the documentation to reference `rustdoc` and of Pandoc.
r? @steveklabnik
Pretty-printed files sometimes start with #![some_feature], which
looks like a shebang line and confuses Windows builds into thinking
they are executables.
This commit leverages the runtime support for DWARF exception info added
in #27210 to enable unwinding by default on 64-bit MSVC. This also additionally
adds a few minor fixes here and there in the test harness and such to get
`make check` entirely passing on 64-bit MSVC:
* The invocation of `maketest.py` now works with spaces/quotes in CC
* debuginfo tests are disabled on MSVC
* A link error for librustc was hacked around (see #27438)
This commit leverages the runtime support for DWARF exception info added
in #27210 to enable unwinding by default on 64-bit MSVC. This also additionally
adds a few minor fixes here and there in the test harness and such to get
`make check` entirely passing on 64-bit MSVC:
* The invocation of `maketest.py` now works with spaces/quotes in CC
* debuginfo tests are disabled on MSVC
* A link error for librustc was hacked around (see #27438)
I have no idea how bors keeps working without this - I can only assume it's some peculiarity of how windows searches for DLLs.
Without this change, running `make check` on windows will not correctly set PATH to include eg. `x86_64-pc-windows-gnu\stage1\bin\rustlib\x86_64-pc-windows-gnu\lib`, and when it tries to run eg. `stage1/test/stdtest-x86_64-pc-windows-gnu.exe`, it will fail because windows can't find the DLLs on which it relies.
It seems to be just a mistake: when the equivalent was added for the branch that deals with unix-like platforms, the windows branch was left unchanged.
This means that we no longer need to ship the `llvm-ar.exe` binary in the MSVC
distribution, and after a snapshot we can remove a good bit of logic from the
makefiles!
Rust's current compilation model makes it impossible on Windows to generate one
object file with a complete and final set of dllexport annotations. This is
because when an object is generated the compiler doesn't actually know if it
will later be included in a dynamic library or not. The compiler works around
this today by flagging *everything* as dllexport, but this has the drawback of
exposing too much.
Thankfully there are alternate methods of specifying the exported surface area
of a dll on Windows, one of which is passing a `*.def` file to the linker which
lists all public symbols of the dynamic library. This commit removes all
locations that add `dllexport` to LLVM variables and instead dynamically
generates a `*.def` file which is passed to the linker. This file will include
all the public symbols of the current object file as well as all upstream
libraries, and the crucial aspect is that it's only used when generating a
dynamic library. When generating an executable this file isn't generated, so all
the symbols aren't exported from an executable.
To ensure that statically included native libraries are reexported correctly,
the previously added support for the `#[linked_from]` attribute is used to
determine the set of FFI symbols that are exported from a dynamic library, and
this is required to get the compiler to link correctly.
This means that we no longer need to ship the `llvm-ar.exe` binary in the MSVC
distribution, and after a snapshot we can remove a good bit of logic from the
makefiles!
This commit removes all morestack support from the compiler which entails:
* Segmented stacks are no longer emitted in codegen.
* We no longer build or distribute libmorestack.a
* The `stack_exhausted` lang item is no longer required
The only current use of the segmented stack support in LLVM is to detect stack
overflow. This is no longer really required, however, because we already have
guard pages for all threads and registered signal handlers watching for a
segfault on those pages (to print out a stack overflow message). Additionally,
major platforms (aka Windows) already don't use morestack.
This means that Rust is by default less likely to catch stack overflows because
if a function takes up more than one page of stack space it won't hit the guard
page. This is what the purpose of morestack was (to catch this case), but it's
better served with stack probes which have more cross platform support and no
runtime support necessary. Until LLVM supports this for all platform it looks
like morestack isn't really buying us much.
cc #16012 (still need stack probes)
Closes#26458 (a drive-by fix to help diagnostics on stack overflow)
New enough find on Linux doesn't support "-perm +..." and suggests
using "-perm /..." instead, but that doesn't work on Windows.
Hopefully all platforms are happy with this expanded version.
As there’s no C++ runtime any more there’s really no point in having anything but Rust tags being made.
I’ve also taken the liberty of excluding the compiler parts of this in the `librust%,,` pattern substitution. Whether or not this is “correct” will depend on whether you want tags for the compiler or for general use. For myself, I want it for general use.
I’m not sure how much people use the tags files anyway. I definitely do, but with Racer existing the tags files aren’t quite so necessary.
I've been baking this out of tree for long enough. This is currently about ~2/5ths the size of TRPL. Time to get it in tree so it can be more widely maintained and scrutinized. I've preserved the whole gruesome history including various rewrites. I can definitely squash these a fair amount if desired. Some random people submitted minor fixes though, so they're mixed in.
Edit: forgot to link to rendered http://cglab.ca/~abeinges/blah/turpl/_book/
Edit2:
To streamline the review process, I'm going to break this into sections that need official "domain expert" approval:
# Summary
* [ ] references.md -- very important, needs work
* [x] Meet Safe and Unsafe: reviewed by @aturon
* [x] Data Layout: reviewed by @arielb1
* [x] Ownership: reviewed by @aturon ( and sorta @nikomatsakis ) -- significantly updated, may need re-r
* [x] Coversions: reviewed by @nrc
* [x] Uninitialized Memory: reviewed by @pnkfelix
* [x] Ownership-Oriented Resource Management: reviewed by @aturon
* [x] Unwinding: reviewed by @alexcrichton
* [x] Concurrency: reviewed by @aturon
* [x] Implementing Vec: r? @huonw
As there’s no C++ runtime any more there’s really no point in having
anything but Rust tags being made.
I’ve also taken the liberty of excluding the compiler parts of this in
the `librust%,,` pattern substitution. Whether or not this is “correct”
will depend on whether you want tags for the compiler or for general
use. For myself, I want it for general use.
I’m not sure how much people use the tags files anyway. I definitely do,
but with Racer existing the tags files aren’t quite so necessary.
This commit moves the IR files in the distribution, rust_try.ll,
rust_try_msvc_64.ll, and rust_try_msvc_32.ll into the compiler from the main
distribution. There's a few reasons for this change:
* LLVM changes its IR syntax from time to time, so it's very difficult to
have these files build across many LLVM versions simultaneously. We'll likely
want to retain this ability for quite some time into the future.
* The implementation of these files is closely tied to the compiler and runtime
itself, so it makes sense to fold it into a location which can do more
platform-specific checks for various implementation details (such as MSVC 32
vs 64-bit).
* This removes LLVM as a build-time dependency of the standard library. This may
end up becoming very useful if we move towards building the standard library
with Cargo.
In the immediate future, however, this commit should restore compatibility with
LLVM 3.5 and 3.6.
We have previously always relied upon an external tool, `ar`, to modify archives
that the compiler produces (staticlibs, rlibs, etc). This approach, however, has
a number of downsides:
* Spawning a process is relatively expensive for small compilations
* Encoding arguments across process boundaries often incurs unnecessary overhead
or lossiness. For example `ar` has a tough time dealing with files that have
the same name in archives, and the compiler copies many files around to ensure
they can be passed to `ar` in a reasonable fashion.
* Most `ar` programs found do **not** have the ability to target arbitrary
platforms, so this is an extra tool which needs to be found/specified when
cross compiling.
The LLVM project has had a tool called `llvm-ar` for quite some time now, but it
wasn't available in the standard LLVM libraries (it was just a standalone
program). Recently, however, in LLVM 3.7, this functionality has been moved to a
library and is now accessible by consumers of LLVM via the `writeArchive`
function.
This commit migrates our archive bindings to no longer invoke `ar` by default
but instead make a library call to LLVM to do various operations. This solves
all of the downsides listed above:
* Archive management is now much faster, for example creating a "hello world"
staticlib is now 6x faster (50ms => 8ms). Linking dynamic libraries also
recently started requiring modification of rlibs, and linking a hello world
dynamic library is now 2x faster.
* The compiler is now one step closer to "hassle free" cross compilation because
no external tool is needed for managing archives, LLVM does the right thing!
This commit does not remove support for calling a system `ar` utility currently.
We will continue to maintain compatibility with LLVM 3.5 and 3.6 looking forward
(so the system LLVM can be used wherever possible), and in these cases we must
shell out to a system utility. All nightly builds of Rust, however, will stop
needing a system `ar`.
This PR was originally going to be a "let's start running tests on MSVC" PR, but it didn't quite get to that point. It instead gets us ~80% of the way there! The steps taken in this PR are:
* Landing pads are turned on by default for 64-bit MSVC. The LLVM support is "good enough" with the caveat the destructor glue is now marked noinline. This was recommended [on the associated bug](https://llvm.org/bugs/show_bug.cgi?id=23884) as a stopgap until LLVM has a better representation for exception handling in MSVC. The consequence of this is that MSVC will have a bit of a perf hit, but there are possible routes we can take if this workaround sticks around for too long.
* The linker (`link.exe`) is now looked up in the Windows Registry if it's not otherwise available in the environment. This improves using the compiler outside of a VS shell (e.g. in a MSYS shell or in a vanilla cmd.exe shell). This also makes cross compiles via Cargo "just work" when crossing between 32 and 64 bit!
* TLS destructors were fixed to start running on MSVC (they previously weren't running at all)
* A few assorted `run-pass` tests were fixed.
* The dependency on the `rust_builtin` library was removed entirely for MSVC to try to prevent any `cl.exe` compiled objects get into the standard library. This should help us later remove any dependence on the CRT by the standard library.
* I re-added `rust_try_msvc_32.ll` for 32-bit MSVC and ensured that landing pads were turned off by default there as well.
Despite landing pads being enabled, there are still *many* failing tests on MSVC. The two major classes I've identified so far are:
* Spurious aborts. It appears that when optimizations are enabled that landing pads aren't always lined up properly, and sometimes an exception being thrown can't find the catch block down the stack, causing the program to abort. I've been working to reduce this test case but haven't been met with great success just yet.
* Parallel codegen does not work on MSVC. Our current strategy is to take the N object files emitted by the N codegen threads and use `ld -r` to assemble them into *one* object file. The MSVC linker, however, does not have this ability, and this will need to be rearchitected to work on MSVC.
I will fix parallel codegen in a future PR, and I'll also be watching LLVM closely to see if the aborts... disappear!
This commit alters the compiler to no longer "just run link.exe" but instead
probe the system's registry to find where the linker is located. The default
library search path (normally found through LIB) is also found through the
registry. This also brings us in line with the default behavior of Clang, and
much of the logic of where to look for information is copied over from Clang as
well. Finally, this commit removes the makefile logic for updating the
environment variables for the compiler, except for stage0 where it's still
necessary.
The motivation for this change is rooted in two positions:
* Not having to set up these environment variables is much less hassle both for
the bootstrap and for running the compiler itself. This means that the
compiler can be run outside of VS shells and be run inside of cmd.exe or a
MSYS shell.
* When dealing with cross compilation, there's not actually a set of environment
variables that can be set for the compiler. This means, for example, if a
Cargo compilation is targeting 32-bit from 64-bit you can't actually set up
one set of environment variables. Having the compiler deal with the logic
instead is generally much more convenient!
This commit modifies the configure script and our makefiles to support building
32-bit MSVC targets. The MSVC toolchain is now parameterized over whether it can
produce a 32-bit or 64-bit binary. The configure script was updated to export
more variables at configure time, and the makefiles were rejiggered to
selectively reexport the relevant environment variables for the applicable
targets they're going to run for.
Now that LLVM has been updated, the only remaining roadblock to implementing
unwinding for MSVC is to fill out the runtime support in `std::rt::unwind::seh`.
This commit does precisely that, fixing up some other bits and pieces along the
way:
* The `seh` unwinding module now uses `RaiseException` to initiate a panic.
* The `rust_try.ll` file was rewritten for MSVC (as it's quite different) and is
located at `rust_try_msvc_64.ll`, only included on MSVC builds for now.
* The personality function for all landing pads generated by LLVM is hard-wired
to `__C_specific_handler` instead of the standard `rust_eh_personality` lang
item. This is required to get LLVM to emit SEH unwinding information instead
of DWARF unwinding information. This also means that on MSVC the
`rust_eh_personality` function is entirely unused (but is defined as it's a
lang item).
More details about how panicking works on SEH can be found in the
`rust_try_msvc_64.ll` or `seh.rs` files, but I'm always open to adding more
comments!
A key aspect of this PR is missing, however, which is that **unwinding is still
turned off by default for MSVC**. There is a [bug in llvm][llvm-bug] which
causes optimizations to inline enough landing pads that LLVM chokes. If the
compiler is optimized at `-O1` (where inlining isn't enabled) then it can
bootstrap with unwinding enabled, but when optimized at `-O2` (inlining is
enabled) then it hits a fatal LLVM error.
[llvm-bug]: https://llvm.org/bugs/show_bug.cgi?id=23884
In #26252 support was added to have prettier paths printed out on failure by not
passing the full path to the source file to the compiler, but instead just a
small relative path. To preserve this relative path across configurations, the
`SREL` variable was used for reconfiguring, but if `SREL` is empty then it will
attempt to run the command `configure` which is distinct from running
`./configure` (e.g. doesn't run the local script).
This commit modifies the `SREL` value to re-run the configure script by setting
it to `./` in the case where `SREL` is empty.
In #26252 support was added to have prettier paths printed out on failure by not
passing the full path to the source file to the compiler, but instead just a
small relative path. To preserve this relative path across configurations, the
`SREL` variable was used for reconfiguring, but if `SREL` is empty then it will
attempt to run the command `configure` which is distinct from running
`./configure` (e.g. doesn't run the local script).
This commit modifies the `SREL` value to re-run the configure script by setting
it to `./` in the case where `SREL` is empty.
musl only creates rlib files for stdlib linking so we need to ignore the `CFG_LIB_GLOB_` setting, otherwise we an error:
```
$ make --debug VERBOSE=1 dist-tar-bins
[...]
Successfully remade target file `prepare-target-x86_64-unknown-linux-gnu-host-x86_64-unknown-linux-gnu-2-dir-x86_64-unknown-linux-gnu'.
File `prepare-target-x86_64-unknown-linux-musl-host-x86_64-unknown-linux-gnu-2-dir-x86_64-unknown-linux-gnu' does not exist.
Must remake target `prepare-target-x86_64-unknown-linux-musl-host-x86_64-unknown-linux-gnu-2-dir-x86_64-unknown-linux-gnu'.
umask 022 && mkdir -p tmp/dist/rustc-1.2.0-dev-x86_64-unknown-linux-gnu-image/lib/rustlib/x86_64-unknown-linux-musl/lib
umask 022 && mkdir -p tmp/dist/rustc-1.2.0-dev-x86_64-unknown-linux-gnu-image/lib/rustlib/x86_64-unknown-linux-gnu/bin
LIB_NAME="liblibc-d8ace771.rlib"; MATCHES=""; if [ -n "$MATCHES" ]; then echo "warning: one or libraries matching Rust library 'liblibc-*.rlib'" && echo " (other than '$LIB_NAME' itself) alre
ady present" && echo " at destination tmp/dist/rustc-1.2.0-dev-x86_64-unknown-linux-gnu-image/lib/rustlib/x86_64-unknown-linux-musl/lib:" && echo $MATCHES ; fi
install -m644 `ls -drt1 x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-musl/lib/liblibc-*.rlib` tmp/dist/rustc-1.2.0-dev-x86_64-unknown-linux-gnu-image/lib/rustlib/x86_64-unk
nown-linux-musl/lib/
LIB_NAME=""; MATCHES=""; if [ -n "$MATCHES" ]; then echo "warning: one or libraries matching Rust library 'libstd-*.so'" && echo " (other than '$LIB_NAME' itself) already present" && echo
" at destination tmp/dist/rustc-1.2.0-dev-x86_64-unknown-linux-gnu-image/lib/rustlib/x86_64-unknown-linux-musl/lib:" && echo $MATCHES ; fi
install -m644 `ls -drt1 x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-*.so` tmp/dist/rustc-1.2.0-dev-x86_64-unknown-linux-gnu-image/lib/rustlib/x86_64-unknow
n-linux-musl/lib/
ls: cannot access x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-*.so: No such file or directory
install: missing destination file operand after ‘tmp/dist/rustc-1.2.0-dev-x86_64-unknown-linux-gnu-image/lib/rustlib/x86_64-unknown-linux-musl/lib/’
Try 'install --help' for more information.
make: *** [prepare-target-x86_64-unknown-linux-musl-host-x86_64-unknown-linux-gnu-2-dir-x86_64-unknown-linux-gnu] Error 1
```
`CFG_INSTALL_ONLY_RLIB_` is provided for this reason and fixes `make install` and `make dist`.
mk: Build crates with relative source file paths
The path we pass to rustc will be visible in panic messages and
backtraces: they will be user visible!
Avoid junk in these paths by passing relative paths to rustc.
For most advanced users, `libcore` or `libstd` in the path will be
a clue to the location -- inside our code, not theirs.
Store both the relative path to the source as well as the absolute.
Use the relative path where it matters, compiling the main crates,
instead of changing all of the build process to cope with relative
paths.
Example output after this patch:
```
$ ./testunwrap
thread '<main>' panicked at 'called `Option::unwrap()` on a `None` value', ../src/libcore/option.rs:362
$ RUST_BACKTRACE=1 ./testunwrap
thread '<main>' panicked at 'called `Option::unwrap()` on a `None` value', ../src/libcore/option.rs:362
stack backtrace:
1: 0x7ff59c1e9956 - sys::backtrace::write::h67a542fd2b201576des
at ../src/libstd/sys/unix/backtrace.rs:158
2: 0x7ff59c1ed5b6 - panicking::on_panic::h3d21c41cdd5c12d41Xw
at ../src/libstd/panicking.rs:58
3: 0x7ff59c1e7b6e - rt::unwind::begin_unwind_inner::h9f3a5440cebb8baeLDw
at ../src/libstd/rt/unwind/mod.rs:273
4: 0x7ff59c1e7f84 - rt::unwind::begin_unwind_fmt::h4fe8a903e0c296b0RCw
at ../src/libstd/rt/unwind/mod.rs:212
5: 0x7ff59c1eced7 - rust_begin_unwind
6: 0x7ff59c22c11a - panicking::panic_fmt::h00b0cd49c98a9220i5B
at ../src/libcore/panicking.rs:64
7: 0x7ff59c22b9e0 - panicking::panic::hf549420c0ee03339P3B
at ../src/libcore/panicking.rs:45
8: 0x7ff59c1e621d - option::Option<T>::unwrap::h501963526474862829
9: 0x7ff59c1e61b1 - main::hb5c91ce92347d1e6eaa
10: 0x7ff59c1f1c18 - rust_try_inner
11: 0x7ff59c1f1c05 - rust_try
12: 0x7ff59c1ef374 - rt::lang_start::h7e51e19c6677cffe5Sw
at ../src/libstd/rt/unwind/mod.rs:147
at ../src/libstd/rt/unwind/mod.rs:130
at ../src/libstd/rt/mod.rs:128
13: 0x7ff59c1e628e - main
14: 0x7ff59b3f6b44 - __libc_start_main
15: 0x7ff59c1e6078 - <unknown>
16: 0x0 - <unknown>
```
The path we pass to rustc will be visible in panic messages and
backtraces: they will be user visible!
Avoid junk in these paths by passing relative paths to rustc.
For most advanced users, `libcore` or `libstd` in the path will be
a clue to the location -- inside our code, not theirs.
Store both the relative path to the source as well as the absolute.
Use the relative path where it matters, compiling the main crates,
instead of changing all of the build process to cope with relative
paths.
Example output after this patch:
```
$ ./testunwrap
thread '<main>' panicked at 'called `Option::unwrap()` on a `None` value', ../src/libcore/option.rs:362
$ RUST_BACKTRACE=1 ./testunwrap
thread '<main>' panicked at 'called `Option::unwrap()` on a `None` value', ../src/libcore/option.rs:362
stack backtrace:
1: 0x7ff59c1e9956 - sys::backtrace::write::h67a542fd2b201576des
at ../src/libstd/sys/unix/backtrace.rs:158
2: 0x7ff59c1ed5b6 - panicking::on_panic::h3d21c41cdd5c12d41Xw
at ../src/libstd/panicking.rs:58
3: 0x7ff59c1e7b6e - rt::unwind::begin_unwind_inner::h9f3a5440cebb8baeLDw
at ../src/libstd/rt/unwind/mod.rs:273
4: 0x7ff59c1e7f84 - rt::unwind::begin_unwind_fmt::h4fe8a903e0c296b0RCw
at ../src/libstd/rt/unwind/mod.rs:212
5: 0x7ff59c1eced7 - rust_begin_unwind
6: 0x7ff59c22c11a - panicking::panic_fmt::h00b0cd49c98a9220i5B
at ../src/libcore/panicking.rs:64
7: 0x7ff59c22b9e0 - panicking::panic::hf549420c0ee03339P3B
at ../src/libcore/panicking.rs:45
8: 0x7ff59c1e621d - option::Option<T>::unwrap::h501963526474862829
9: 0x7ff59c1e61b1 - main::hb5c91ce92347d1e6eaa
10: 0x7ff59c1f1c18 - rust_try_inner
11: 0x7ff59c1f1c05 - rust_try
12: 0x7ff59c1ef374 - rt::lang_start::h7e51e19c6677cffe5Sw
at ../src/libstd/rt/unwind/mod.rs:147
at ../src/libstd/rt/unwind/mod.rs:130
at ../src/libstd/rt/mod.rs:128
13: 0x7ff59c1e628e - main
14: 0x7ff59b3f6b44 - __libc_start_main
15: 0x7ff59c1e6078 - <unknown>
16: 0x0 - <unknown>
```
Right now the distribution tarball for MSVC only includes the *.dll files for
the supporting libraries, but not the corresponding *.lib files which allow
actually linking to the dll. This means that the current MSVC nightlies cannot
produce dynamically linked binaries as the *.lib files are not available to link
against.
This commit modifies the `LIB_GLOB` used to copy the files around to include the
`lib` variant of the `dll`.
