This fixes the case where we try to re-build & re-install rust to the
same prefix (without uninstalling) while using an llvm-root that is the
same as the prefix.
Without this, builds like that fail with:
'error: multiple dylib candidates for `std` found'
See https://github.com/jmesmon/meta-rust/issues/6 for some details.
May also be related to #20342.
This commit removes all morestack support from the compiler which entails:
* Segmented stacks are no longer emitted in codegen.
* We no longer build or distribute libmorestack.a
* The `stack_exhausted` lang item is no longer required
The only current use of the segmented stack support in LLVM is to detect stack
overflow. This is no longer really required, however, because we already have
guard pages for all threads and registered signal handlers watching for a
segfault on those pages (to print out a stack overflow message). Additionally,
major platforms (aka Windows) already don't use morestack.
This means that Rust is by default less likely to catch stack overflows because
if a function takes up more than one page of stack space it won't hit the guard
page. This is what the purpose of morestack was (to catch this case), but it's
better served with stack probes which have more cross platform support and no
runtime support necessary. Until LLVM supports this for all platform it looks
like morestack isn't really buying us much.
cc #16012 (still need stack probes)
Closes#26458 (a drive-by fix to help diagnostics on stack overflow)
The path we pass to rustc will be visible in panic messages and
backtraces: they will be user visible!
Avoid junk in these paths by passing relative paths to rustc.
For most advanced users, `libcore` or `libstd` in the path will be
a clue to the location -- inside our code, not theirs.
Store both the relative path to the source as well as the absolute.
Use the relative path where it matters, compiling the main crates,
instead of changing all of the build process to cope with relative
paths.
Example output after this patch:
```
$ ./testunwrap
thread '<main>' panicked at 'called `Option::unwrap()` on a `None` value', ../src/libcore/option.rs:362
$ RUST_BACKTRACE=1 ./testunwrap
thread '<main>' panicked at 'called `Option::unwrap()` on a `None` value', ../src/libcore/option.rs:362
stack backtrace:
1: 0x7ff59c1e9956 - sys::backtrace::write::h67a542fd2b201576des
at ../src/libstd/sys/unix/backtrace.rs:158
2: 0x7ff59c1ed5b6 - panicking::on_panic::h3d21c41cdd5c12d41Xw
at ../src/libstd/panicking.rs:58
3: 0x7ff59c1e7b6e - rt::unwind::begin_unwind_inner::h9f3a5440cebb8baeLDw
at ../src/libstd/rt/unwind/mod.rs:273
4: 0x7ff59c1e7f84 - rt::unwind::begin_unwind_fmt::h4fe8a903e0c296b0RCw
at ../src/libstd/rt/unwind/mod.rs:212
5: 0x7ff59c1eced7 - rust_begin_unwind
6: 0x7ff59c22c11a - panicking::panic_fmt::h00b0cd49c98a9220i5B
at ../src/libcore/panicking.rs:64
7: 0x7ff59c22b9e0 - panicking::panic::hf549420c0ee03339P3B
at ../src/libcore/panicking.rs:45
8: 0x7ff59c1e621d - option::Option<T>::unwrap::h501963526474862829
9: 0x7ff59c1e61b1 - main::hb5c91ce92347d1e6eaa
10: 0x7ff59c1f1c18 - rust_try_inner
11: 0x7ff59c1f1c05 - rust_try
12: 0x7ff59c1ef374 - rt::lang_start::h7e51e19c6677cffe5Sw
at ../src/libstd/rt/unwind/mod.rs:147
at ../src/libstd/rt/unwind/mod.rs:130
at ../src/libstd/rt/mod.rs:128
13: 0x7ff59c1e628e - main
14: 0x7ff59b3f6b44 - __libc_start_main
15: 0x7ff59c1e6078 - <unknown>
16: 0x0 - <unknown>
```
Special thanks to @retep998 for the [excellent writeup](https://github.com/rust-lang/rfcs/issues/1061) of tasks to be done and @ricky26 for initially blazing the trail here!
# MSVC Support
This goal of this series of commits is to add MSVC support to the Rust compiler
and build system, allowing it more easily interoperate with Visual Studio
installations and native libraries compiled outside of MinGW.
The tl;dr; of this change is that there is a new target of the compiler,
`x86_64-pc-windows-msvc`, which will not interact with the MinGW toolchain at
all and will instead use `link.exe` to assemble output artifacts.
## Why try to use MSVC?
With today's Rust distribution, when you install a compiler on Windows you also
install `gcc.exe` and a number of supporting libraries by default (this can be
opted out of). This allows installations to remain independent of MinGW
installations, but it still generally requires native code to be linked with
MinGW instead of MSVC. Some more background can also be found in #1768 about the
incompatibilities between MinGW and MSVC.
Overall the current installation strategy is quite nice so long as you don't
interact with native code, but once you do the usage of a MinGW-based `gcc.exe`
starts to get quite painful.
Relying on a nonstandard Windows toolchain has also been a long-standing "code
smell" of Rust and has been slated for remedy for quite some time now. Using a
standard toolchain is a great motivational factor for improving the
interoperability of Rust code with the native system.
## What does it mean to use MSVC?
"Using MSVC" can be a bit of a nebulous concept, but this PR defines it as:
* The build system for Rust will build as much code as possible with the MSVC
compiler, `cl.exe`.
* The build system will use native MSVC tools for managing archives.
* The compiler will link all output with `link.exe` instead of `gcc.exe`.
None of these are currently implemented today, but all are required for the
compiler to fluently interoperate with MSVC.
## How does this all work?
At the highest level, this PR adds a new target triple to the Rust compiler:
x86_64-pc-windows-msvc
All logic for using MSVC or not is scoped within this triple and code can
conditionally build for MSVC or MinGW via:
#[cfg(target_env = "msvc")]
It is expected that auto builders will be set up for MSVC-based compiles in
addition to the existing MinGW-based compiles, and we will likely soon start
shipping MSVC nightlies where `x86_64-pc-windows-msvc` is the host target triple
of the compiler.
# Summary of changes
Here I'll explain at a high level what many of the changes made were targeted
at, but many more details can be found in the commits themselves. Many thanks to
@retep998 for the excellent writeup in rust-lang/rfcs#1061 and @rick26 for a lot
of the initial proof-of-concept work!
## Build system changes
As is probably expected, a large chunk of this PR is changes to Rust's build
system to build with MSVC. At a high level **it is an explicit non goal** to
enable building outside of a MinGW shell, instead all Makefile infrastructure we
have today is retrofitted with support to use MSVC instead of the standard MSVC
toolchain. Some of the high-level changes are:
* The configure script now detects when MSVC is being targeted and adds a number
of additional requirements about the build environment:
* The `--msvc-root` option must be specified or `cl.exe` must be in PATH to
discover where MSVC is installed. The compiler in use is also required to
target x86_64.
* Once the MSVC root is known, the INCLUDE/LIB environment variables are
scraped so they can be reexported by the build system.
* CMake is required to build LLVM with MSVC (and LLVM is also configured with
CMake instead of the normal configure script).
* jemalloc is currently unconditionally disabled for MSVC targets as jemalloc
isn't a hard requirement and I don't know how to build it with MSVC.
* Invocations of a C and/or C++ compiler are now abstracted behind macros to
appropriately call the underlying compiler with the correct format of
arguments, for example there is now a macro for "assemble an archive from
objects" instead of hard-coded invocations of `$(AR) crus liboutput.a ...`
* The output filenames for standard libraries such as morestack/compiler-rt are
now "more correct" on windows as they are shipped as `foo.lib` instead of
`libfoo.a`.
* Rust targets can now depend on native tools provided by LLVM, and as you'll
see in the commits the entire MSVC target depends on `llvm-ar.exe`.
* Support for custom arbitrary makefile dependencies of Rust targets has been
added. The MSVC target for `rustc_llvm` currently requires a custom `.DEF`
file to be passed to the linker to get further linkages to complete.
## Compiler changes
The modifications made to the compiler have so far largely been minor tweaks
here and there, mostly just adding a layer of abstraction over whether MSVC or a
GNU-like linker is being used. At a high-level these changes are:
* The section name for metadata storage in dynamic libraries is called `.rustc`
for MSVC-based platorms as section names cannot contain more than 8
characters.
* The implementation of `rustc_back::Archive` was refactored, but the
functionality has remained the same.
* Targets can now specify the default `ar` utility to use, and for MSVC this
defaults to `llvm-ar.exe`
* The building of the linker command in `rustc_trans:🔙:link` has been
abstracted behind a trait for the same code path to be used between GNU and
MSVC linkers.
## Standard library changes
Only a few small changes were required to the stadnard library itself, and only
for minor differences between the C runtime of msvcrt.dll and MinGW's libc.a
* Some function names for floating point functions have leading underscores, and
some are not present at all.
* Linkage to the `advapi32` library for crypto-related functions is now
explicit.
* Some small bits of C code here and there were fixed for compatibility with
MSVC's cl.exe compiler.
# Future Work
This commit is not yet a 100% complete port to using MSVC as there are still
some key components missing as well as some unimplemented optimizations. This PR
is already getting large enough that I wanted to draw the line here, but here's
a list of what is not implemented in this PR, on purpose:
## Unwinding
The revision of our LLVM submodule [does not seem to implement][llvm] does not
support lowering SEH exception handling on the Windows MSVC targets, so
unwinding support is not currently implemented for the standard library (it's
lowered to an abort).
[llvm]: https://github.com/rust-lang/llvm/blob/rust-llvm-2015-02-19/lib/CodeGen/Passes.cpp#L454-L461
It looks like, however, that upstream LLVM has quite a bit more support for SEH
unwinding and landing pads than the current revision we have, so adding support
will likely just involve updating LLVM and then adding some shims of our own
here and there.
## dllimport and dllexport
An interesting part of Windows which MSVC forces our hand on (and apparently
MinGW didn't) is the usage of `dllimport` and `dllexport` attributes in LLVM IR
as well as native dependencies (in C these correspond to
`__declspec(dllimport)`).
Whenever a dynamic library is built by MSVC it must have its public interface
specified by functions tagged with `dllexport` or otherwise they're not
available to be linked against. This poses a few problems for the compiler, some
of which are somewhat fundamental, but this commit alters the compiler to attach
the `dllexport` attribute to all LLVM functions that are reachable (e.g. they're
already tagged with external linkage). This is suboptimal for a few reasons:
* If an object file will never be included in a dynamic library, there's no need
to attach the dllexport attribute. Most object files in Rust are not destined
to become part of a dll as binaries are statically linked by default.
* If the compiler is emitting both an rlib and a dylib, the same source object
file is currently used but with MSVC this may be less feasible. The compiler
may be able to get around this, but it may involve some invasive changes to
deal with this.
The flipside of this situation is that whenever you link to a dll and you import
a function from it, the import should be tagged with `dllimport`. At this time,
however, the compiler does not emit `dllimport` for any declarations other than
constants (where it is required), which is again suboptimal for even more
reasons!
* Calling a function imported from another dll without using `dllimport` causes
the linker/compiler to have extra overhead (one `jmp` instruction on x86) when
calling the function.
* The same object file may be used in different circumstances, so a function may
be imported from a dll if the object is linked into a dll, but it may be
just linked against if linked into an rlib.
* The compiler has no knowledge about whether native functions should be tagged
dllimport or not.
For now the compiler takes the perf hit (I do not have any numbers to this
effect) by marking very little as `dllimport` and praying the linker will take
care of everything. Fixing this problem will likely require adding a few
attributes to Rust itself (feature gated at the start) and then strongly
recommending static linkage on Windows! This may also involve shipping a
statically linked compiler on Windows instead of a dynamically linked compiler,
but these sorts of changes are pretty invasive and aren't part of this PR.
## CI integration
Thankfully we don't need to set up a new snapshot bot for the changes made here as our snapshots are freestanding already, we should be able to use the same snapshot to bootstrap both MinGW and MSVC compilers (once a new snapshot is made from these changes).
I plan on setting up a new suite of auto bots which are testing MSVC configurations for now as well, for now they'll just be bootstrapping and not running tests, but once unwinding is implemented they'll start running all tests as well and we'll eventually start gating on them as well.
---
I'd love as many eyes on this as we've got as this was one of my first interactions with MSVC and Visual Studio, so there may be glaring holes that I'm missing here and there!
cc @retep998, @ricky26, @vadimcn, @klutzy
r? @brson
The compiler will require that `llvm-ar.exe` be available for MSVC-targeting
builds (more comments on this soon), so this commit adds support for targets to
depend on LLVM tools. The `core` library for MSVC depends on `llvm-ar.exe` which
will be copied into place for the target before the compiler starts to run.
Note that these targets all depend on `llvm-config.exe` to ensure that they're
built before they're attempted to be copied.
This commit modifies the makefiles to enable building LLVM with cmake and Visual
Studio to generate an LLVM that targets MSVC. Rust's configure script requires
cmake to be installed when targeting MSVC and will configure LLVM with cmake
instead of the normal `./configure` script LLVM provides. The build will then
run cmake to execute the build instead of the normal `make`.
Currently `make clean-llvm` isn't supported on MSVC as I can't figure out how to
run a "clean" target for the Visual Studio files.
Previously libmorestack.a and libcompiler-rt.a were installed, but link.exe
looks for morestack.lib and compiler-rt.lib by default, so we need to install
these with the correct name
The code takes a prefix of the MD5 hash of the version string.
Since the hash command differs across GNU and BSD platforms, we scan for
the right one in the configure script.
Closes#25007
The code takes a prefix of the MD5 hash of the version string.
Since the hash command differs across GNU and BSD platforms, we scan for
the right one in the configure script.
Closes#25007
This commit adds support to the makefiles, configuration script, and build
system to understand MUSL. This is broken up into a few parts:
* Any target of the form `*-musl` requires the `--musl-root` option to
`./configure` which will indicate the root of the MUSL installation. It is
also expected that there is a libunwind build inside of that installation
built against that MUSL.
* Objects from MUSL are copied into the build tree for Rust to be statically
linked into the appropriate Rust library.
* Objects for binary startup and shutdown are included in each Rust installation
by default for MUSL. This requires MUSL to only be installed on the machine
compiling rust. Only a linker will be necessary for compiling against MUSL on
a target machine.
Eventually a MUSL and/or libunwind build may be integrated by default into the
build but for now they are just always assumed to exist externally.
Instead of rustc-1.0.0-beta-$triple.tar.gz, betas will be named
rustc-beta-$triple.tar.gz. This will give betas a stable download
URL, prevent old artifacts from accumulating in the dist server's
root directory, and not require the website to be updated every
beta.
As a tradeoff, it will be harder to download previous betas because
they will need to be located in the archives.
LLVM_LIBDIR_<triple> is only defined for host triples, not target triples.
FWIW, the same is true for LLVM_STDCPP_RUSTFLAGS_<triple>, where we
explicitly define it as empty when --enable-llvm-static-stdcpp is not
specified, but it's still undefined for cross-compiled triples.
This partially implements the feature staging described in the
[release channel RFC][rc]. It does not yet fully conform to the RFC as
written, but does accomplish its goals sufficiently for the 1.0 alpha
release.
It has three primary user-visible effects:
* On the nightly channel, use of unstable APIs generates a warning.
* On the beta channel, use of unstable APIs generates a warning.
* On the beta channel, use of feature gates generates a warning.
Code that does not trigger these warnings is considered 'stable',
modulo pre-1.0 bugs.
Disabling the warnings for unstable APIs continues to be done in the
existing (i.e. old) style, via `#[allow(...)]`, not that specified in
the RFC. I deem this marginally acceptable since any code that must do
this is not using the stable dialect of Rust.
Use of feature gates is itself gated with the new 'unstable_features'
lint, on nightly set to 'allow', and on beta 'warn'.
The attribute scheme used here corresponds to an older version of the
RFC, with the `#[staged_api]` crate attribute toggling the staging
behavior of the stability attributes, but the user impact is only
in-tree so I'm not concerned about having to make design changes later
(and I may ultimately prefer the scheme here after all, with the
`#[staged_api]` crate attribute).
Since the Rust codebase itself makes use of unstable features the
compiler and build system to a midly elaborate dance to allow it to
bootstrap while disobeying these lints (which would otherwise be
errors because Rust builds with `-D warnings`).
This patch includes one significant hack that causes a
regression. Because the `format_args!` macro emits calls to unstable
APIs it would trigger the lint. I added a hack to the lint to make it
not trigger, but this in turn causes arguments to `println!` not to be
checked for feature gates. I don't presently understand macro
expansion well enough to fix. This is bug #20661.
Closes#16678
[rc]: https://github.com/rust-lang/rfcs/blob/master/text/0507-release-channels.md
Adds a new 'beta cycle' variable that can be appended to the '-beta' version label, e.g. '-beta1'. Changes the version label for the beta channel temporarily to 'alpha'. Changes the artifact name of the beta channel to contain the version number instead of just being called 'beta'. The beta cycle number is currently set to 1.
The impact of this is that the first alphas will be called '1.0.0-alpha1' and the artifacts will also be called '1.0.0-alpha1-*.tar.gz'. We could alternately leave out the cycle number if we are confident there will be only one alpha cycle.
r? @alexcrichton cc @nikomatsakis @huonw
This is a collection of misc issues I've run into while adding bindir & libdir support that aren't really bindir & libdir specific.
While I continue to fiddle with bindir and libdir bugs, I figured these might be useful for others to have merged.
We add CFG_LLVM_TARGET_$(target) (which can be defined in any of the
mk/cfg/* files) and supply a default to the plain target name
CFG_LLVM_TARGET mirrors the value of llvm_target (aka llvm-target) in
the librustc_back runtime target specification.
not in hardcoded libdir path. If there was no LIBDIR provided
during configuration fallback to hardcoded paths.
Thanks to Jan Niklas Hasse for solution and to Alex Crichton for improvements.
Closes#11671
Commit bec2ee77f7 started quoting paths
discovered as part of the `probe` function, which includes git. The
`make` `wildcard` function appears to be incompatible with quoted
paths so this check in the makefile now fails. Employing `wildcard`
here appears to only re-verify that git actually exists, which the
configure script already did, so I've just removed it.
Additionally, with the quoted paths the `subst` function should no
longer be needed, so I've removed it as well.
Closes#18771
Right now the windows nightlies are failing because they're encountering a
linker error when producing stage3 libs. The stage3 libs aren't actually used in
general, and we primarily just want to generate a static stage3 binary, not
static stage3 dylibs.
If a dylib is being produced, the compiler will now first check to see if it can
be created entirely statically before falling back to dynamic dependencies. This
behavior can be overridden with `-C prefer-dynamic`.
Due to the alteration in behavior, this is a breaking change. Any previous users
relying on dylibs implicitly maximizing dynamic dependencies should start
passing `-C prefer-dynamic` to compilations.
Closes#18499
[breaking-change]
Removes all target-specific knowledge from rustc. Some targets have changed
during this, but none of these should be very visible outside of
cross-compilation. The changes make our targets more consistent.
iX86-unknown-linux-gnu is now only available as i686-unknown-linux-gnu. We
used to accept any value of X greater than 1. i686 was released in 1995, and
should encompass the bare minimum of what Rust supports on x86 CPUs.
The only two windows targets are now i686-pc-windows-gnu and
x86_64-pc-windows-gnu.
The iOS target has been renamed from arm-apple-ios to arm-apple-darwin.
A complete list of the targets we accept now:
arm-apple-darwin
arm-linux-androideabi
arm-unknown-linux-gnueabi
arm-unknown-linux-gnueabihf
i686-apple-darwin
i686-pc-windows-gnu
i686-unknown-freebsd
i686-unknown-linux-gnu
mips-unknown-linux-gnu
mipsel-unknown-linux-gnu
x86_64-apple-darwin
x86_64-unknown-freebsd
x86_64-unknown-linux-gnu
x86_64-pc-windows-gnu
Closes#16093
[breaking-change]
If a dylib is being produced, the compiler will now first check to see if it can
be created entirely statically before falling back to dynamic dependencies. This
behavior can be overridden with `-C prefer-dynamic`.
Due to the alteration in behavior, this is a breaking change. Any previous users
relying on dylibs implicitly maximizing dynamic dependencies should start
passing `-C prefer-dynamic` to compilations.
Closes#18499
[breaking-change]
Enable parallel codegen (2 units) by default when --opt-level is 0 or 1. This
gives a minor speedup on large crates (~10%), with only a tiny slowdown (~2%)
for small ones (which usually build in under a second regardless). The current
default (no parallelization) is used when the user requests optimization
(--opt-level 2 or 3), and when the user has enabled LTO (which is incompatible
with parallel codegen).
This commit also changes the rust build system to use parallel codegen
when appropriate. This means codegen-units=4 for stage0 always, and
also for stage1 and stage2 when configured with --disable-optimize.
(Other settings use codegen-units=1 for stage1 and stage2, to get
maximum performance for release binaries.) The build system also sets
codegen-units=1 for compiletest tests (compiletest does its own
parallelization) and uses the same setting as stage2 for crate tests.
Adds a new configure flag, --release-channel, which determines how the version
number should be augmented with a release label, as well as how the distribution
artifacts will be named. This is entirely for use by the build automation.
--release-channel can be either 'source', 'nightly', 'beta', or 'stable'.
Here's a summary of the affect of these values on version number and
artifact naming, respectively:
* source - '0.12.0-pre', 'rust-0.12.0-pre-...'
* nightly - '0.12.0-nightly', 'rust-nightly-...'
* beta - '0.12.0-beta', 'rust-beta-...'
* stable - '0.12.0', 'rust-0.12.0-...'
Per http://discuss.rust-lang.org/t/rfc-impending-changes-to-the-release-process/508/1
The performance hit from these checks is significant, but unoptimized
builds are already incredibly slow. Enabling these checks results in
better test coverage since there are bots doing unoptimized builds, and
the cost is relatively small in the context of an unoptimized build.
This also allows using `JEMALLOC_FLAGS` to override the default
configure flags.
This commit disables rustc's emission of rpath attributes into dynamic libraries
and executables by default. The functionality is still preserved, but it must
now be manually enabled via a `-C rpath` flag.
This involved a few changes to the local build system:
* --disable-rpath is now the default configure option
* Makefiles now prefer our own LD_LIBRARY_PATH over the user's LD_LIBRARY_PATH
in order to support building rust with rust already installed.
* The compiletest program was taught to correctly pass through the aux dir as a
component of LD_LIBRARY_PATH in more situations.
The major impact of this change is that neither rustdoc nor rustc will work
out-of-the-box in all situations because they are dynamically linked. It must be
arranged to ensure that the libraries of a rust installation are part of the
LD_LIBRARY_PATH. The default installation paths for all platforms ensure this,
but if an installation is in a nonstandard location, then configuration may be
necessary.
Additionally, for all developers of rustc, it will no longer be possible to run
$target/stageN/bin/rustc out-of-the-box. The old behavior can be regained
through the `--enable-rpath` option to the configure script.
This change brings linux/mac installations in line with windows installations
where rpath is not possible.
Closes#11747
[breaking-change]
The compiler will no longer insert a hash or version into a filename by default.
Instead, all output is simply based off the crate name being compiled. For
example, a crate name of `foo` would produce the following outputs:
* bin => foo
* rlib => libfoo.rlib
* dylib => libfoo.{so,dylib} or foo.dll
* staticlib => libfoo.a
The old behavior has been moved behind a new codegen flag,
`-C extra-filename=<hash>`. For example, with the "extra filename" of `bar` and
a crate name of `foo`, the following outputs would be generated:
* bin => foo (same old behavior)
* rlib => libfoobar.rlib
* dylib => libfoobar.{so,dylib} or foobar.dll
* staticlib => libfoobar.a
The makefiles have been altered to pass a hash by default to invocations of
`rustc` so all installed rust libraries will have a hash in their filename. This
is done because the standard libraries are intended to be installed into
privileged directories such as /usr/local. Additionally, it involves very few
build system changes!
RFC: 0035-remove-crate-id
[breaking-change]
This commit disables rustc's emission of rpath attributes into dynamic libraries
and executables by default. The functionality is still preserved, but it must
now be manually enabled via a `-C rpath` flag.
This involved a few changes to the local build system:
* --disable-rpath is now the default configure option
* Makefiles now prefer our own LD_LIBRARY_PATH over the user's LD_LIBRARY_PATH
in order to support building rust with rust already installed.
* The compiletest program was taught to correctly pass through the aux dir as a
component of LD_LIBRARY_PATH in more situations.
The major impact of this change is that neither rustdoc nor rustc will work
out-of-the-box in all situations because they are dynamically linked. It must be
arranged to ensure that the libraries of a rust installation are part of the
LD_LIBRARY_PATH. The default installation paths for all platforms ensure this,
but if an installation is in a nonstandard location, then configuration may be
necessary.
Additionally, for all developers of rustc, it will no longer be possible to run
$target/stageN/bin/rustc out-of-the-box. The old behavior can be regained
through the `--enable-rpath` option to the configure script.
This change brings linux/mac installations in line with windows installations
where rpath is not possible.
Closes#11747
[breaking-change]
This involved a few changes to the local build system:
* Makefiles now prefer our own LD_LIBRARY_PATH over the user's LD_LIBRARY_PATH
in order to support building rust with rust already installed.
* The compiletest program was taught to correctly pass through the aux dir as a
component of LD_LIBRARY_PATH in more situations.
This change was spliced out of #14832 to consist of just the fixes to running
tests without an rpath setting embedded in executables.
Two line summary: Distinguish HOST_RPATH and TARGET_RPATH; added
RPATH_LINK_SEARCH; skip tests broken in stage1; general cleanup.
`HOST_RPATH_VAR$(1)_T_$(2)_H_$(3)` and `TARGET_RPATH_VAR$(1)_T_$(2)_H_$(3)`
both match the format of the old `RPATH_VAR$(1)_T_$(2)_H_$(3)` (which
is still being set the same way that it was before, to one of either
HOST/TARGET depending on what stage we are building). Namely, the format
is <XXX>_RPATH_VAR = "<LD_LIB_PATH_ENVVAR>=<COLON_SEP_PATH_ENTRIES>"
What this commit does:
* Pass both of the (newly introduced) HOST and TARGET rpath setup vars
to `maketest.py`
* Update `maketest.py` to no longer update the LD_LIBRARY_PATH itself
Instead, it passes along the HOST and TARGET rpath setup vars in
environment variables `HOST_RPATH_ENV` and `TARGET_RPATH_ENV`
* Also, pass the current stage number to maketest.py; it in turn
passes it (via an env var) to run-make tests.
This allows the run-make tests to selectively change behavior
(e.g. turn themselves off) to deal with incompatibilities with
e.g. stage1.
* Cleanup: Distinguish in tools.mk between the command to run (`RUN`)
and the file to generate to drive that command (`RUN_BINFILE`). The
main thing this enables is that `RUN` can now setup the
`TARGET_RPATH_ENV` without having to dirty up the runner code in
each of the `run-make` Makefiles.
* Cleanup: Factored out commands to delete dylib/rlib into
REMOVE_DYLIBS/REMOVE_RLIBS.
There were places where we were only calling `rm $(call DYLIB,foo)`
even though we really needed to get rid of the whole glob (at least
based on alex's findings on #13753 that removing the symlink does not
suffice).
Therefore rather than peppering the code with the awkward
`rm $(TMPDIR)/$(call DYLIB_GLOB,foo)`, I instead introduced a common
`REMOVE_DYLIBS` user function that expands into that when called.
After I adding an analogous `REMOVE_RLIBS`, I changed all of the
existing calls that rm dylibs or rlibs to use these routines
instead.
Note that the latter is not a true refactoring since I may have
changed cases where it was our intent to only remove the sym-link.
(But if that is the case, then we need to more deeply investigate
alex's findings on #13753 where the system was still dynamically
loading up the non-symlinked libraries that it finds on the load
path.)
* Added RPATH_LINK_SEARCH command and use it on Linux.
On some platforms, namely Linux, when you have libboot.so that has
its internal rpath set (to e.g. $(ORIGIN)/path/to/HOSTDIR), the
linker still complains when you do the link step and it does not
know where to find libraries that libboot.so depends upon that live
in HOSTDIR (think e.g. librustuv.so).
As far as I can tell, the GNU linker will consult the
LD_LIBRARY_PATH as part of the linking process to find such
libraries. But if you want to be more careful and not override
LD_LIBRARY_PATH for the `gcc` invocation, then you need some other
way to tell the linker where it can find the libraries that
libboot.so needs. The solution to this on Linux is the
`-Wl,-rpath-link` command line option.
However, this command line option does not exist on Mac OS X, (which
appears to be figuring out how to resolve the libboot.dylib
dependency by some other means, perhaps by consulting the rpath
setting within libboot.dylib).
So, in order to abstract over this distinction, I added the
RPATH_LINK_SEARCH macro to the run-make infrastructure and added
calls to it where necessary to get Linux working. On architectures
other than Linux, the macro expands to nothing.
* Disable miscellaneous tests atop stage1.
* An especially interesting instance of the previous bullet point:
Excuse regex from doing rustdoc tests atop stage1.
This was a (nearly-) final step to get `make check-stage1` working
again.
The use of a special-case check for regex here is ugly but is
analogous other similar checks for regex such as the one that landed
in PR #13844.
The way this is written, the user will get a reminder that
doc-crate-regex is being skipped whenever their rules attempt to do
the crate documentation tests. This is deliberate: I want people
running `make check-stage1` to be reminded about which cases are
being skipped. (But if such echo noise is considered offensive, it
can obviously be removed.)
* Got windows working with the above changes.
This portion of the commit is a cleanup revision of the (previously
mentioned on try builds) re-architecting of how the LD_LIBRARY_PATH
setup and extension is handled in order to accommodate Windows' (1.)
use of `$PATH` for that purpose and (2.) use of spaces in `$PATH`
entries (problematic for make and for interoperation with tools at
the shell).
* In addition, since the code has been rearchitected to pass the
HOST_RPATH_DIR/TARGET_RPATH_DIR rather than a whole sh
environment-variable setting command, there is no need to for the
convert_path_spec calls in maketest.py, which in fact were put in
place to placate Windows but were now causing the Windows builds to
fail. Instead we just convert the paths to absolute paths just like
all of the other path arguments.
Also, note for makefile hackers: apparently you cannot quote operands
to `ifeq` in Makefile (or at least, you need to be careful about
adding them, e.g. to only one side).
After removing absolute rpaths, cross compile builds (notably the nightly
builders) broke. This is because the RPATH was pointing at an empty directory
because only the rustc binary is copied over, not all of the target libraries.
This modifies the cross compile logic to fixup the rpath of the stage0
cross-compiled rustc to point to where it came from.
Concerns have been raised about using absolute rpaths in #11746, and this is the
first step towards not relying on rpaths at all. The only current use case for
an absolute rpath is when a non-installed rust builds an executable that then
moves from is built location. The relative rpath back to libstd and absolute
rpath to the installation directory still remain (CFG_PREFIX).
Closes#11746
Rebasing of #12754
