`thorin` is a Rust implementation of a DWARF packaging utility that
supports reading DWARF objects from archive files (i.e. rlibs) and
therefore is better suited for integration into rustc.
Signed-off-by: David Wood <david.wood@huawei.com>
In #79570, `-Z split-dwarf-kind={none,single,split}` was replaced by `-C
split-debuginfo={off,packed,unpacked}`. `-C split-debuginfo`'s packed
and unpacked aren't exact parallels to single and split, respectively.
On Unix, `-C split-debuginfo=packed` will put debuginfo into object
files and package debuginfo into a DWARF package file (`.dwp`) and
`-C split-debuginfo=unpacked` will put debuginfo into dwarf object files
and won't package it.
In the initial implementation of Split DWARF, split mode wrote sections
which did not require relocation into a DWARF object (`.dwo`) file which
was ignored by the linker and then packaged those DWARF objects into
DWARF packages (`.dwp`). In single mode, sections which did not require
relocation were written into object files but ignored by the linker and
were not packaged. However, both split and single modes could be
packaged or not, the primary difference in behaviour was where the
debuginfo sections that did not require link-time relocation were
written (in a DWARF object or the object file).
This commit re-introduces a `-Z split-dwarf-kind` flag, which can be
used to pick between split and single modes when `-C split-debuginfo` is
used to enable Split DWARF (either packed or unpacked).
Signed-off-by: David Wood <david.wood@huawei.com>
Actually set IMAGE_SCN_LNK_REMOVE for .rmeta
The code intended to set the IMAGE_SCN_LNK_REMOVE flag for the
.rmeta section, however the value of this flag was set to zero.
Instead use the actual value provided by the object crate.
This dates back to the original introduction of this code in
PR #84449, so we were never setting this flag. As I'm not on
Windows, I'm not sure whether that means we were embedding .rmeta
into executables, or whether the section ended up getting stripped
for some other reason.
Allow loading LLVM plugins with both legacy and new pass manager
Opening a draft PR to get feedback and start discussion on this feature. There is already a codegen option `passes` which allow giving a list of LLVM pass names, however we currently can't use a LLVM pass plugin (as described here : https://llvm.org/docs/WritingAnLLVMPass.html), the only available passes are the LLVM built-in ones.
The proposed modification would be to add another codegen option `pass-plugins`, which can be set with a list of paths to shared library files. These libraries are loaded using the LLVM function `PassPlugin::Load`, which calls the expected symbol `lvmGetPassPluginInfo`, and register the pipeline parsing and optimization callbacks.
An example usage with a single plugin and 3 passes would look like this in the `.cargo/config`:
```toml
rustflags = [
"-C", "pass-plugins=/tmp/libLLVMPassPlugin",
"-C", "passes=pass1 pass2 pass3",
]
```
This would give the same functionality as the opt LLVM tool directly integrated in rust build system.
Additionally, we can also not specify the `passes` option, and use a plugin which inserts passes in the optimization pipeline, as one could do using clang.
The code intended to set the IMAGE_SCN_LNK_REMOVE flag for the
.rmeta section, however the value of this flag was set to zero.
Instead use the actual value provided by the object crate.
This dates back to the original introduction of this code in
PR #84449, so we were never setting this flag. As I'm not on
Windows, I'm not sure whether that means we were embedding .rmeta
into executables, or whether the section ended up getting stripped
for some other reason.
Explicitly set no ELF flags for .rustc section
For a data section, the object crate will set the SHF_ALLOC by default, which is exactly what we don't want. Explicitly set sh_flags to zero to avoid this.
I checked with `objdump -h` that this produces the right flags for ELF.
Fixes#92013.
Remove `SymbolStr`
This was originally proposed in https://github.com/rust-lang/rust/pull/74554#discussion_r466203544. As well as removing the icky `SymbolStr` type, it allows the removal of a lot of `&` and `*` occurrences.
Best reviewed one commit at a time.
r? `@oli-obk`
Rollup of 7 pull requests
Successful merges:
- #91566 (Apply path remapping to DW_AT_GNU_dwo_name when producing split DWARF)
- #91926 (Remove `in_band_lifetimes` from `rustc_metadata`)
- #91931 (Remove `in_band_lifetimes` from `rustc_codegen_llvm`)
- #92024 (rustc_codegen_llvm: Give each codegen unit a unique DWARF name on all platforms, not just Apple ones.)
- #92037 (Use a const ParamEnv when in default_method_body_is_const)
- #92047 (Set `RUST_BACKTRACE=0` when running location-detail tests)
- #92050 (Add a space and 2 grave accents )
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
For a data section, the object crate will set the SHF_ALLOC by
default, which is exactly what we don't want. Explicitly set
sh_flags to zero to avoid this.
Apply path remapping to DW_AT_GNU_dwo_name when producing split DWARF
`--remap-path-prefix` doesn't apply to paths to `.o` (in case of packed) or `.dwo` (in case of unpacked) files in `DW_AT_GNU_dwo_name`. GCC also has this bug https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91888
DF_ORIGIN flag signifies that the object being loaded may make reference to the $ORIGIN substitution string.
Some implementations are just ignoring DF_ORIGIN and do substitution for $ORIGIN if present (whatever DF_ORIGIN pr
Set the flag inconditionally if rpath is wanted.
Remove `in_band_lifetimes` from `rustc_codegen_ssa`
See #91867 for more information.
In `compiler/rustc_codegen_ssa/src/coverageinfo/map.rs`, there are several functions with an explicit `'a` lifetime but only a single `&'a self` parameter. These lifetimes should be redundant given lifetime elision, unless the existential `impl Iterator` has weird issues regarding that. Should the redundant lifetimes be removed?
We already use the object crate for generating uncompressed .rmeta
metadata object files. This switches the generation of compressed
.rustc object files to use the object crate as well. These have
slightly different requirements in that .rmeta should be completely
excluded from any final compilation artifacts, while .rustc should
be part of shared objects, but not loaded into memory.
The primary motivation for this change is #90326: In LLVM 14, the
current way of setting section flags (and in particular, preventing
the setting of SHF_ALLOC) will no longer work. There are other ways
we could work around this, but switching to the object crate seems
like the most elegant, as we already use it for .rmeta, and as it
makes this independent of the codegen backend. In particular, we
don't need separate handling in codegen_llvm and codegen_gcc.
codegen_cranelift should be able to reuse the implementation as
well, though I have omitted that here, as it is not based on
codegen_ssa.
This change mostly extracts the existing code for .rmeta handling
to allow using it for .rustc as well, and adjust the codegen
infrastructure to handle the metadata object file separately: We
no longer create a backend-specific module for it, and directly
produce the compiled module instead.
This does not fix#90326 by itself yet, as .llvmbc will need to be
handled separately.
Fix ld64 flags
- The `-exported_symbols_list` argument appears to be malformed for `ld64` (if you are not going through `clang`).
- The `-dynamiclib` argument isn't support for `ld64`. It should be guarded behind a compiler flag.
These problems are fixed by these changes. I have also refactored the way linker arguments are generated to be ld/compiler agnostic and therefore less error prone.
These changes are necessary to support cross-compilation to darwin targets.
Leave -Z strip available temporarily as an alias, to avoid breaking
cargo until cargo transitions to using -C strip. (If the user passes
both, the -C version wins.)
This commit refactors linker argument generation to leverage a helper
function that abstracts away details governing how these arguments are
transformed and provided to the linker.
This fixes the misuse of the `-exported_symbols_list` when an ld-like
linker is used rather than a compiler. A compiler would expect
`-Wl,-exported_symbols_list,path` but ld would expect
`-exported_symbols_list` and `path` as two seperate arguments. Prior
to this change, an ld-like linker was given
`-exported_symbols_list,path`.
Linker arguments must transformed when Rust is interacting with the
linker through a compiler. This commit introduces a helper function
that abstracts away details of this transformation.
Record more artifact sizes during self-profiling.
This PR adds artifact size recording for
- "linked artifacts" (executables, RLIBs, dylibs, static libs)
- object files
- dwo files
- assembly files
- crate metadata
- LLVM bitcode files
- LLVM IR files
- codegen unit size estimates
Currently the identifiers emitted for these are hard-coded as string literals. Is it worth adding constants to https://github.com/rust-lang/measureme/blob/master/measureme/src/rustc.rs instead? We don't do that for query names and the like -- but artifact kinds might be more stable than query names.
Don't abort compilation after giving a lint error
The only reason to use `abort_if_errors` is when the program is so broken that either:
1. later passes get confused and ICE
2. any diagnostics from later passes would be noise
This is never the case for lints, because the compiler has to be able to deal with `allow`-ed lints.
So it can continue to lint and compile even if there are lint errors.
Closes https://github.com/rust-lang/rust/issues/82761. This is a WIP because I have a feeling it will exit with 0 even if there were lint errors; I don't have a computer that can build rustc locally at the moment.
The only reason to use `abort_if_errors` is when the program is so broken that either:
1. later passes get confused and ICE
2. any diagnostics from later passes would be noise
This is never the case for lints, because the compiler has to be able to deal with `allow`-ed lints.
So it can continue to lint and compile even if there are lint errors.
In https://reviews.llvm.org/D71059 LLVM 11, the time trace profiler was
extended to support multiple threads.
`timeTraceProfilerInitialize` creates a thread local profiler instance.
When a thread finishes `timeTraceProfilerFinishThread` moves a thread
local instance into a global collection of instances. Finally when all
codegen work is complete `timeTraceProfilerWrite` writes data from the
current thread local instance and the instances in global collection
of instances.
Previously, the profiler was intialized on a single thread only. Since
this thread performs no code generation on its own, the resulting
profile was empty.
Update LLVM codegen to initialize & finish time trace profiler on each
code generation thread.
This performs a substitution of code following the pattern:
let <id> = if let <pat> = ... { identity } else { ... : ! };
To simplify it to:
let <pat> = ... { identity } else { ... : ! };
By adopting the let_else feature.
rustc_driver: Enable the `WARN` log level by default
This commit changes the `tracing_subscriber` initialization in
`rustc_driver` so that the `WARN` verbosity level is enabled by default
when the `RUSTC_LOG` env variable is empty. If the `RUSTC_LOG` env
variable is set, the filter string in the environment variable is
honored, instead.
Fixes#76824Closes#89623
cc ``@eddyb,`` ``@oli-obk``
On macOS, make strip="symbols" not pass any options to strip
This makes the output with `strip="symbols"` match the result of just
calling `strip` on the output binary, minimizing the size of the binary.
This largely involves implementing the options debug-info-for-profiling
and profile-sample-use and forwarding them on to LLVM.
AutoFDO can be used on x86-64 Linux like this:
rustc -O -Cdebug-info-for-profiling main.rs -o main
perf record -b ./main
create_llvm_prof --binary=main --out=code.prof
rustc -O -Cprofile-sample-use=code.prof main.rs -o main2
Now `main2` will have feedback directed optimization applied to it.
The create_llvm_prof tool can be obtained from this github repository:
https://github.com/google/autofdoFixes#64892.
Fix handling of +whole-archive native link modifier.
This PR fixes a bug in `add_upstream_native_libraries` that led to the `+whole-archive` modifier being ignored when linking in native libs.
~~Note that the PR does not address the situation when `+whole-archive` is combined with `+bundle`.~~
`@wesleywiser's` commit adds validation code that turns combining `+whole-archive` with `+bundle` into an error.
Fixes https://github.com/rust-lang/rust/issues/88085.
r? `@petrochenkov`
cc `@wesleywiser` `@gcoakes`
Make `-Z gcc-ld=lld` work for Apple targets
`-Z gcc-ld=lld` was introduced in #85961. It does not work on Macos because lld needs be either named `ld64` or passed `-flavor darwin` as the first two arguments in order to select the Mach-O flavor. Rust invokes cc (=clang) on Macos for linking which calls `ld` as linker binary and not `ld64`, so just creating an `ld64` binary and modifying the search path with `-B` does not work.
In order to solve this patch does:
* Set the `lld_flavor` for all Apple-derived targets to `LldFlavor::Ld64`. As far as I can see this actually works towards fixing `-Xlinker=rust-lld` as all those targets use the Mach-O object format.
* Copy/hardlink rust-lld to the gcc-ld subdirectory as ld64 next to ld.
* If `-Z gcc-ld=lld` is used and the target lld flavor is Ld64 add `-fuse-ld=/path/to/ld64` to the linker invocation.
Fixes#86945.
Link the static libraries with "-bundle" modifier from upstream rust crate
right after linking this rust crate. Some linker such as GNU linker
`ld.bdf` treat order of linking as order of dependency. After this change,
static libraries with "-bundle" modifier is linked in the same order as
"+bundle" modifier. So we can change the value of "bundle" modifier without
causing linking error.
Don't use gc-sections with profile-generate.
When building with profile-generate don't call gc_sections as this can
can sometimes strip out profile data. This missing information in the
prof files can then result in missing functions when using the profile
information.
#78226
r? `@Mark-Simulacrum`
When building with profile-generate request that metadata is kept
during the gc_sections call, as this can sometimes strip out profile
data.
This missing information in the prof files can then result in missing
functions when using the profile information.
Add suggestions for "undefined reference" link errors
This adds a suggestion for "undefined reference to ..." linking errors to install or specify the location to an external library. Since there is no defined error format for linkers, we just check if there was a failure and if that failure contains the string `undefined reference to`. This also makes it impossible to test this, since the output depends on the system linker. The output now looks like:
```
error: linking with `cc` failed: exit status: 1
|
= note: "cc" "-m64" "linking_failure.linking_failure.7rcbfp3g-cgu.0.rcgu.o" "linking_failure.linking_failure.7rcbfp3g-cgu.1.rcgu.o" "linking_failure.linking_failure.7rcbfp3g-cgu.2.rcgu.o" "linking_failure.linking_failure.7rcbfp3g-cgu.3.rcgu.o" "linking_failure.linking_failure.7rcbfp3g-cgu.4.rcgu.o" "linking_failure.linking_failure.7rcbfp3g-cgu.5.rcgu.o" "linking_failure.linking_failure.7rcbfp3g-cgu.6.rcgu.o" "linking_failure.linking_failure.7rcbfp3g-cgu.7.rcgu.o" "linking_failure.linking_failure.7rcbfp3g-cgu.8.rcgu.o" "linking_failure.53u64zklswtfazes.rcgu.o" "-Wl,--as-needed" "-L" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,--start-group" "-Wl,-Bstatic" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-01ce3ba5c629d02f.rlib" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-f1f2102409186354.rlib" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-1e8b6b56a999f838.rlib" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-d0e93eb4e14f1d19.rlib" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-1d7e39d75d082b43.rlib" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-ade42e945045b261.rlib" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-1a65064fccf4ebc1.rlib" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-4d699c310fdfe72d.rlib" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-1cafa68a696ec800.rlib" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-e9f1c8c4dab2f046.rlib" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-ecc1a743be25c7f7.rlib" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-e074031c4b66b6b6.rlib" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-9aa6ed9f1d3bfd53.rlib" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-7862bf96c2250ca0.rlib" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-f02ce0dc7895b5fd.rlib" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-3af9c60917570521.rlib" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-ca16fc7bb3645684.rlib" "-Wl,--end-group" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-d8e1a5b7299604cc.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/home/smit/rustc-dev/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "linking_failure" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs"
= note: /usr/bin/ld: linking_failure.linking_failure.7rcbfp3g-cgu.3.rcgu.o: in function `linking_failure::main':
linking_failure.7rcbfp3g-cgu.3:(.text._ZN15linking_failure4main17h52b6e3052e444479E+0x3): undefined reference to `doesnt_exist_thiwthwfyl'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
= help: some `extern` functions couldn't be found; you may need to install or specify the path to some dependencies
= note: use the -L flag to specify the library lookup path
= note: use the cargo:rustc-link-search directive to specify the library lookup path with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#rustc-link-search)
error: aborting due to previous error
```
ignore test if rust-lld not found
create ld -> rust-lld symlink at build time instead of run time
for testing in ci
copy instead of symlinking
remove linux check
test for linker, suggestions from bjorn3
fix overly restrictive lld matcher
use -Zgcc-ld flag instead of -Clinker-flavor
refactor code adding lld to gcc path
revert ci changes
suggestions from petrochenkov
rename gcc_ld to gcc-ld in dirs
As reported in the stabilization issue, MacOS' linker doesn't support the `-s` and `-S` flags to strip symbols anymore. However, the os ships a separated tool to perform these operations.
This change allows the compiler to use that tool after a target has been compiled to strip symbols.
For rationale, see: https://github.com/rust-lang/rust/issues/72110#issuecomment-641169818
For option selection, see: https://www.unix.com/man-page/osx/1/strip/
Signed-off-by: David Calavera <david.calavera@gmail.com>
Remove the install prefix from the rpath set when using -Crpath
It was broken anyway for rustup installs and nobody seems to have noticed.
Fixes https://github.com/rust-lang/rust/issues/82392
Unify duplicate linker_and_flavor methods in rustc_codegen_{cranelift,ssa}.
The two methods were exactly the same so this removes the cranelift copy. This will help make sure both they don't get out of sync.
Driver improvements
This PR contains a couple of cleanups for the driver and a few small improvements for the custom codegen backend interface. It also implements `--version` and `-Cpasses=list` support for custom codegen backends.
- Combine all native library arguments together, to simplify potential support for library deduplication and similar things
- Split arguments into order-independent and order-dependent, to define more precisely what (pre,post,late)-link-args mean
Tweak wasm_base target spec to indicate linker is not GNU and update linker inferring logic for wasm-ld.
Reported via [Zulip](https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/wasi.20linker.20unknown.20argument.3A.20--as-needed): we try passing `--as-needed` to the linker if it's GNU ld which `wasm-ld` is not. Usually this isn't an issue for wasm as we would use the WasmLd linker driver but because the linker in question (`wasm32-unknown-wasi-wasm-ld`) ended with `-ld` our linker inferring [logic](f64503eb55/compiler/rustc_codegen_ssa/src/back/link.rs (L957-L1040)) used the `GccLinker` implementations. (UPD: The linker inferring logic actually didn't apply in this case because the linker is actually invoked through gcc in the reported issue. But it's still worth updating the logic I think.)
This change then has 2 parts:
1. Update wasm_base target spec to indicate `linker_is_gnu: false` plus a few additions of `target.is_like_wasm` to handle flags `wasm-ld` does in fact support.
2. Improve the linker detection logic to properly determine the correct flavor of wasm linker we're using when we can.
We need to add the new `target.is_like_wasm` branches to handle the case where the "linker" used could be something like clang which would then under the hood call wasm-ld.
Preserve metadata w/ Solaris-like linkers.
#84468 moved the `-zignore` linker flag from the `gc_sections` method to `add_as_needed` which is more accurate but Solaris-style linkers will also end up removing an unreferenced ELF sections [1]. This had the unfortunate side effect of causing the `.rustc` section (which has the metada) to be removed which could cause issues when trying to link against the resulting crates or use proc macros.
Since the `-zignore` is positional, we fix this by moving the metadata objects to before the flag.
[1] Specifically a section is considered unreferenced if:
* The section is allocatable
* No other sections bind to (relocate) to this section
* The section provides no global symbols
https://docs.oracle.com/cd/E19683-01/817-3677/6mj8mbtbs/index.html#chapter4-19
Partial support for raw-dylib linkage
First cut of functionality for issue #58713: add support for `#[link(kind = "raw-dylib")]` on `extern` blocks in lib crates compiled to .rlib files. Does not yet support `#[link_name]` attributes on functions, or the `#[link_ordinal]` attribute, or `#[link(kind = "raw-dylib")]` on `extern` blocks in bin crates; I intend to publish subsequent PRs to fill those gaps. It's also not yet clear whether this works for functions in `extern "stdcall"` blocks; I also intend to investigate that shortly and make any necessary changes as a follow-on PR.
This implementation calls out to an LLVM function to construct the actual `.idata` sections as temporary `.lib` files on disk and then links those into the generated .rlib.
BPF target support
This adds `bpfel-unknown-none` and `bpfeb-unknown-none`, two new no_std targets that generate little and big endian BPF. The approach taken is very similar to the cuda target, where `TargetOptions::obj_is_bitcode` is enabled and code generation is done by the linker.
I added the targets to `dist-various-2`. There are [some tests](https://github.com/alessandrod/bpf-linker/tree/main/tests/assembly) in bpf-linker and I'm planning to add more. Those are currently not ran as part of rust CI.
This does not yet support #[link_name] attributes on functions, the #[link_ordinal]
attribute, #[link(kind = "raw-dylib")] on extern blocks in bin crates, or
stdcall functions on 32-bit x86.
This commit updates how rustc compiler metadata is stored in rlibs.
Previously metadata was stored as a raw file that has the same format as
`--emit metadata`. After this commit, however, the metadata is encoded
into a small object file which has one section which is the contents of
the metadata.
The motivation for this commit is to fix a common case where #83730
arises. The problem is that when rustc crates a `dylib` crate type it
needs to include entire rlib files into the dylib, so it passes
`--whole-archive` (or the equivalent) to the linker. The problem with
this, though, is that the linker will attempt to read all files in the
archive. If the metadata file were left as-is (today) then the linker
would generate an error saying it can't read the file. The previous
solution was to alter the rlib just before linking, creating a new
archive in a temporary directory which has the metadata file removed.
This problem from before this commit is now removed if the metadata file
is stored in an object file that the linker can read. The only caveat we
have to take care of is to ensure that the linker never actually
includes the contents of the object file into the final output. We apply
similar tricks as the `.llvmbc` bytecode sections to do this.
This involved changing the metadata loading code a bit, namely updating
some of the LLVM C APIs used to use non-deprecated ones and fiddling
with the lifetimes a bit to get everything to work out. Otherwise though
this isn't intended to be a functional change really, only that metadata
is stored differently in archives now.
This should end up fixing #83730 because by default dylibs will no
longer have their rlib dependencies "altered" meaning that
split-debuginfo will continue to have valid paths pointing at the
original rlibs. (note that we still "alter" rlibs if LTO is enabled to
remove Rust object files and we also "alter" for the #[link(cfg)]
feature, but that's rarely used).
Closes#83730
For extern providers, both provide and provide_extern are called.
wasm_import_module_map is already provided in provide, so it doesn't
need to be provided in provide_extern.
Only pass --[no-]gc-sections if linker is GNU ld.
Fixes a regression from #84468 where linking now fails with solaris linkers. LinkerFlavor::Gcc does not always mean GNU ld specifically. And in the case of at least the solaris ld in illumos, that flag is unrecognized and will cause the linking step to fail.
Even though removing the `is_like_solaris` branch from `gc_sections` in #84468 made sense as `-z ignore/record` are more analogous to the `--[no-]-as-needed` flags, it inadvertently caused solaris linkers to be passed the `--gc-sections` flag. So let's just change it to be more explicit about when we pass those flags.
Remove CrateNum parameter for queries that only work on local crate
The pervasive `CrateNum` parameter is a remnant of the multi-crate rustc idea.
Using `()` as query key in those cases avoids having to worry about the validity of the query key.
Use the object crate for metadata reading
This allows sharing the metadata reader between cg_llvm, cg_clif and other codegen backends.
This is not currently useful for rlib reading with cg_spirv ([rust-gpu](https://github.com/EmbarkStudios/rust-gpu/)) as it uses tar rather than ar as .rlib format, but it is useful for dylib reading required for loading proc macros. (cc `@eddyb)`
The object crate is already trusted as dependency of libstd through backtrace. As far as I know it supports reading all object file formats used by targets for which we support rust dylibs with crate metadata, but I am not certain. If this happens to not be the case, I could keep using LLVM for reading dylib metadata.
Marked as WIP for a perf run and as it is based on #83637.
LinkerFlavor::Gcc does not always mean GNU ld specifically. And in the
case of at least the solaris ld in illumos, that flag is unrecognized
and will cause the linking step to fail.
Under some conditions, the toolchain will produce a sequence of linker
arguments that result in a NEEDED list that puts libc before libgcc_s;
e.g.,
[0] NEEDED 0x2046ba libc.so.1
[1] NEEDED 0x204723 libm.so.2
[2] NEEDED 0x204736 libsocket.so.1
[3] NEEDED 0x20478b libumem.so.1
[4] NEEDED 0x204763 libgcc_s.so.1
Both libc and libgcc_s provide an unwinder implementation, but libgcc_s
provides some extra symbols upon which Rust directly depends. If libc
is first in the NEEDED list we will find some of those symbols in libc
but others in libgcc_s, resulting in undefined behaviour as the two
implementations do not use compatible interior data structures.
This solution is not perfect, but is the simplest way to produce correct
binaries on illumos for now.
Fix#84467 linker_args with --target=sparcv9-sun-solaris
Trying to cross-compile for sparcv9-sun-solaris
getting a error message for -zignore
Introduced when -z -ignore was seperated here
22d0ab0
No formatting done
Reproduce
``` bash
rustup target add sparcv9-sun-solaris
cargo new --bin hello && cd hello && cargo run --target=sparcv9-sun-solaris
```
config.toml
[target.sparcv9-sun-solaris]
linker = "gcc"
This commit implements both the native linking modifiers infrastructure
as well as an initial attempt at the individual modifiers from the RFC.
It also introduces a feature flag for the general syntax along with
individual feature flags for each modifier.
Moved -z ignore to add_as_needed
Trying to cross-compile for sparcv9-sun-solaris
getting an error message for -zignore
Introduced when -z -ignore was separated here
22d0ab0
No formatting done
Reproduce
``` bash
rustup target add sparcv9-sun-solaris
cargo new --bin hello && cd hello && cargo run --target=sparcv9-sun-solaris
```
config.toml
[target.sparcv9-sun-solaris]
linker = "gcc"
Use FromStr trait for number option parsing
Replace `parse_uint` with generic `parse_number` based on `FromStr`.
Use it for parsing inlining threshold to avoid casting later.
Add an Mmap wrapper to rustc_data_structures
This wrapper implements StableAddress and falls back to directly reading the file on wasm32.
Taken from #83640, which I will close due to the perf regression.
Adjusted LLVM codegen for code compiled with `-Zinstrument-coverage` to
address multiple, somewhat related issues.
Fixed a significant flaw in prior coverage solution: Every counter
generated a new counter variable, but there should have only been one
counter variable per function. This appears to have bloated .profraw
files significantly. (For a small program, it increased the size by
about 40%. I have not tested large programs, but there is anecdotal
evidence that profraw files were way too large. This is a good fix,
regardless, but hopefully it also addresses related issues.
Fixes: #82144
Invalid LLVM coverage data produced when compiled with -C opt-level=1
Existing tests now work up to at least `opt-level=3`. This required a
detailed analysis of the LLVM IR, comparisons with Clang C++ LLVM IR
when compiled with coverage, and a lot of trial and error with codegen
adjustments.
The biggest hurdle was figuring out how to continue to support coverage
results for unused functions and generics. Rust's coverage results have
three advantages over Clang's coverage results:
1. Rust's coverage map does not include any overlapping code regions,
making coverage counting unambiguous.
2. Rust generates coverage results (showing zero counts) for all unused
functions, including generics. (Clang does not generate coverage for
uninstantiated template functions.)
3. Rust's unused functions produce minimal stubbed functions in LLVM IR,
sufficient for including in the coverage results; while Clang must
generate the complete LLVM IR for each unused function, even though
it will never be called.
This PR removes the previous hack of attempting to inject coverage into
some other existing function instance, and generates dedicated instances
for each unused function. This change, and a few other adjustments
(similar to what is required for `-C link-dead-code`, but with lower
impact), makes it possible to support LLVM optimizations.
Fixes: #79651
Coverage report: "Unexecuted instantiation:..." for a generic function
from multiple crates
Fixed by removing the aforementioned hack. Some "Unexecuted
instantiation" notices are unavoidable, as explained in the
`used_crate.rs` test, but `-Zinstrument-coverage` has new options to
back off support for either unused generics, or all unused functions,
which avoids the notice, at the cost of less coverage of unused
functions.
Fixes: #82875
Invalid LLVM coverage data produced with crate brotli_decompressor
Fixed by disabling the LLVM function attribute that forces inlining, if
`-Z instrument-coverage` is enabled. This attribute is applied to
Rust functions with `#[inline(always)], and in some cases, the forced
inlining breaks coverage instrumentation and reports.
Set codegen thread names
Set names on threads spawned during codegen. Various debugging and profiling tools can take advantage of this to show a more useful identifier for threads.
For example, gdb will show thread names in `info threads`:
```
(gdb) info threads
Id Target Id Frame
1 Thread 0x7fffefa7ec40 (LWP 2905) "rustc" __pthread_clockjoin_ex (threadid=140737214134016, thread_return=0x0, clockid=<optimized out>, abstime=<optimized out>, block=<optimized out>)
at pthread_join_common.c:145
2 Thread 0x7fffefa7b700 (LWP 2957) "rustc" 0x00007ffff125eaa8 in llvm::X86_MC::initLLVMToSEHAndCVRegMapping(llvm::MCRegisterInfo*) ()
from /home/wesley/.rustup/toolchains/stage1/lib/librustc_driver-f866439e29074957.so
3 Thread 0x7fffeef0f700 (LWP 3116) "rustc" futex_wait_cancelable (private=0, expected=0, futex_word=0x7fffe8602ac8) at ../sysdeps/nptl/futex-internal.h:183
* 4 Thread 0x7fffeed0e700 (LWP 3123) "rustc" rustc_codegen_ssa:🔙:write::spawn_work (cgcx=..., work=...) at /home/wesley/code/rust/rust/compiler/rustc_codegen_ssa/src/back/write.rs:1573
6 Thread 0x7fffe113b700 (LWP 3150) "opt foof.7rcbfp" 0x00007ffff2940e62 in llvm::CallGraph::populateCallGraphNode(llvm::CallGraphNode*) ()
from /home/wesley/.rustup/toolchains/stage1/lib/librustc_driver-f866439e29074957.so
8 Thread 0x7fffe0d39700 (LWP 3158) "opt foof.7rcbfp" 0x00007fffefe8998e in malloc_consolidate (av=av@entry=0x7ffe2c000020) at malloc.c:4492
9 Thread 0x7fffe0f3a700 (LWP 3162) "opt foof.7rcbfp" 0x00007fffefef27c4 in __libc_open64 (file=0x7fffe0f38608 "foof.foof.7rcbfp3g-cgu.6.rcgu.o", oflag=524865) at ../sysdeps/unix/sysv/linux/open64.c:48
(gdb)
```
and Windows Performance Analyzer will also show this information when profiling:
rustc_codegen_ssa: tune codegen according to available concurrency
This change tunes ahead-of-time codegening according to the amount of
concurrency available, rather than according to the number of CPUs on
the system. This can lower memory usage by reducing the number of
compiled LLVM modules in memory at once, particularly across several
rustc instances.
Previously, each rustc instance would assume that it should codegen
ahead of time to meet the demand of number-of-CPUs workers. But often, a
rustc instance doesn't have nearly that much concurrency available to
it, because the concurrency availability is split, via the jobserver,
across all active rustc instances spawned by the driving cargo process,
and is further limited by the `-j` flag argument. Therefore, each rustc
might have had several times the number of LLVM modules in memory than
it really needed to meet demand. If the modules were large, the effect
on memory usage would be noticeable.
With this change, the required amount of ahead-of-time codegen scales up
with the actual number of workers running within a rustc instance. Note
that the number of workers running can be less than the actual
concurrency available to a rustc instance. However, if more concurrency
is actually available, workers are spun up quickly as job tokens are
acquired, and the ahead-of-time codegen scales up quickly as well.
Set path of the compile unit to the source directory
As part of the effort to implement split dwarf debug info, we ended up
setting the compile unit location to the output directory rather than
the source directory. Furthermore, it seems like we failed to remap the
prefixes for this as well!
The desired behaviour is to instead set the `DW_AT_GNU_dwo_name` to a
path relative to compiler's working directory. This still allows
debuggers to find the split dwarf files, while not changing the
behaviour of the code that is compiling with regular debug info, and not
changing the compiler's behaviour with regards to reproducibility.
Fixes#82074
cc `@alexcrichton` `@davidtwco`
remove redundant option/result wrapping of return values
If a function always returns `Ok(something)`, we can return `something` directly and remove the corresponding error handling in the callers.
clippy::unnecessary_wraps
Add new `rustc` target for Arm64 machines that can target the iphonesimulator
This PR lands a new target (`aarch64-apple-ios-sim`) that targets arm64 iphone simulator, previously unreachable from Apple Silicon machines.
resolves#81632
r? `@shepmaster`
This change tunes ahead-of-time codegening according to the amount of
concurrency available, rather than according to the number of CPUs on
the system. This can lower memory usage by reducing the number of
compiled LLVM modules in memory at once, particularly across several
rustc instances.
Previously, each rustc instance would assume that it should codegen
ahead of time to meet the demand of number-of-CPUs workers. But often, a
rustc instance doesn't have nearly that much concurrency available to
it, because the concurrency availability is split, via the jobserver,
across all active rustc instances spawned by the driving cargo process,
and is further limited by the `-j` flag argument. Therefore, each rustc
might have had several times the number of LLVM modules in memory than
it really needed to meet demand. If the modules were large, the effect
on memory usage would be noticeable.
With this change, the required amount of ahead-of-time codegen scales up
with the actual number of workers running within a rustc instance. Note
that the number of workers running can be less than the actual
concurrency available to a rustc instance. However, if more concurrency
is actually available, workers are spun up quickly as job tokens are
acquired, and the ahead-of-time codegen scales up quickly as well.
In the backend we may want to remove certain temporary files, but in
certain other situations these files might not be produced in the first
place. We don't exactly care about that, and the intent is really that
these files are gone after a certain point in the backend.
Here we unify the backend file removing calls to use `ensure_removed`
which will attempt to delete a file, but will not fail if it does not
exist (anymore).
The tradeoff to this approach is, of course, that we may miss instances
were we are attempting to remove files at wrong paths due to some bug –
compilation would silently succeed but the temporary files would remain
there somewhere.
As part of the effort to implement split dwarf debug info, we ended up
setting the compile unit location to the output directory rather than
the source directory. Furthermore, it seems like we failed to remap the
prefixes for this as well!
The desired behaviour is to instead set the `DW_AT_GNU_dwo_name` to a
path relative to compiler's working directory. This still allows
debuggers to find the split dwarf files, while not changing the
behaviour of the code that is compiling with regular debug info, and not
changing the compiler's behaviour with regards to reproducibility.
Fixes#82074
Use -target when linking binaries for Mac Catalyst
When running `rustc` with `-target x86_64-apple-ios-macabi`, the linker
eventually gets run with `-arch x86_64`, because the linker back end splits the
LLVM target triple and uses the first token as the target architecture. However,
this does not work for the Mac Catalyst ABI, which is a separate target from
Darwin.
Specifying the full target triple with `-target` allows Mac Catalyst binaries to
link and run.
closes#80202
This commit adds a new stable codegen option to rustc,
`-Csplit-debuginfo`. The old `-Zrun-dsymutil` flag is deleted and now
subsumed by this stable flag. Additionally `-Zsplit-dwarf` is also
subsumed by this flag but still requires `-Zunstable-options` to
actually activate. The `-Csplit-debuginfo` flag takes one of
three values:
* `off` - This indicates that split-debuginfo from the final artifact is
not desired. This is not supported on Windows and is the default on
Unix platforms except macOS. On macOS this means that `dsymutil` is
not executed.
* `packed` - This means that debuginfo is desired in one location
separate from the main executable. This is the default on Windows
(`*.pdb`) and macOS (`*.dSYM`). On other Unix platforms this subsumes
`-Zsplit-dwarf=single` and produces a `*.dwp` file.
* `unpacked` - This means that debuginfo will be roughly equivalent to
object files, meaning that it's throughout the build directory
rather than in one location (often the fastest for local development).
This is not the default on any platform and is not supported on Windows.
Each target can indicate its own default preference for how debuginfo is
handled. Almost all platforms default to `off` except for Windows and
macOS which default to `packed` for historical reasons.
Some equivalencies for previous unstable flags with the new flags are:
* `-Zrun-dsymutil=yes` -> `-Csplit-debuginfo=packed`
* `-Zrun-dsymutil=no` -> `-Csplit-debuginfo=unpacked`
* `-Zsplit-dwarf=single` -> `-Csplit-debuginfo=packed`
* `-Zsplit-dwarf=split` -> `-Csplit-debuginfo=unpacked`
Note that `-Csplit-debuginfo` still requires `-Zunstable-options` for
non-macOS platforms since split-dwarf support was *just* implemented in
rustc.
There's some more rationale listed on #79361, but the main gist of the
motivation for this commit is that `dsymutil` can take quite a long time
to execute in debug builds and provides little benefit. This means that
incremental compile times appear that much worse on macOS because the
compiler is constantly running `dsymutil` over every single binary it
produces during `cargo build` (even build scripts!). Ideally rustc would
switch to not running `dsymutil` by default, but that's a problem left
to get tackled another day.
Closes#79361
Fix sysroot option not being honored across rustc
Change link_sanitizer_runtime() to check if the sanitizer library exists in the specified/session sysroot, and if it doesn't exist, use the default sysroot. (See #79253.)
Change link_sanitizer_runtime() to check if the sanitizer library exists
in the specified/session sysroot, and if it doesn't exist, use the
default sysroot.
This commit adds a Split DWARF compare mode to compiletest so that
debuginfo tests are also tested using Split DWARF in split mode (and
manually in single mode).
Signed-off-by: David Wood <david@davidtw.co>
This commit implements Split DWARF support, wiring up the flag (added in
earlier commits) to the modified FFI wrapper (also from earlier
commits).
Signed-off-by: David Wood <david@davidtw.co>
This commit removes the `TargetMachineFactory` struct and adds a
`TargetMachineFactoryFn` type alias which is used everywhere that the
previous, long type was used.
Signed-off-by: David Wood <david@davidtw.co>
This commit changes some comments to documentation comments so that
they can be read on the generated rustdoc.
Signed-off-by: David Wood <david@davidtw.co>
Warn if `dsymutil` returns an error code
This checks the error code returned by `dsymutil` and warns if it failed. It
also provides the stdout and stderr logs from `dsymutil`, similar to the native
linker step.
I tried to think of ways to test this change, but so far I haven't found a good way, as you'd likely need to inject some nonsensical args into `dsymutil` to induce failure, which feels too artificial to me. Also, https://github.com/rust-lang/rust/issues/79361 suggests Rust is on the verge of disabling `dsymutil` by default, so perhaps it's okay for this change to be untested. In any case, I'm happy to add a test if someone sees a good approach.
Fixes https://github.com/rust-lang/rust/issues/78770
This checks the error code returned by `dsymutil` and warns if it failed. It
also provides the stdout and stderr logs from `dsymutil`, similar to the native
linker step.
Fixes https://github.com/rust-lang/rust/issues/78770
with an eye on merging `TargetOptions` into `Target`.
`TargetOptions` as a separate structure is mostly an implementation detail of `Target` construction, all its fields logically belong to `Target` and available from `Target` through `Deref` impls.
The wrapper type led to tons of target.target
across the compiler. Its ptr_width field isn't
required any more, as target_pointer_width
is already present in parsed form.
Preparation for a subsequent change that replaces
rustc_target::config::Config with its wrapped Target.
On its own, this commit breaks the build. I don't like making
build-breaking commits, but in this instance I believe that it
makes review easier, as the "real" changes of this PR can be
seen much more easily.
Result of running:
find compiler/ -type f -exec sed -i -e 's/target\.target\([)\.,; ]\)/target\1/g' {} \;
find compiler/ -type f -exec sed -i -e 's/target\.target$/target/g' {} \;
find compiler/ -type f -exec sed -i -e 's/target.ptr_width/target.pointer_width/g' {} \;
./x.py fmt
Rename target_pointer_width to pointer_width because it is already
member of the Target struct.
The compiler supports only three valid values for target_pointer_width:
16, 32, 64. Thus it can safely be turned into an int.
This means less allocations and clones as well as easier handling of the type.
