Avoid a branch on key being local for queries that use the same local and extern providers
Currently based on https://github.com/rust-lang/rust/pull/85810 as it slightly conflicts with it. Only the last two commits are new.
Add -Z no-unique-section-names to reduce ELF header bloat.
This change adds a new compiler flag that can help reduce the size of ELF binaries that contain many functions.
By default, when enabling function sections (which is the default for most targets), the LLVM backend will generate different section names for each function. For example, a function `func` would generate a section called `.text.func`. Normally this is fine because the linker will merge all those sections into a single one in the binary. However, starting with [LLVM 12](https://github.com/llvm/llvm-project/commit/ee5d1a04), the backend will also generate unique section names for exception handling, resulting in thousands of `.gcc_except_table.*` sections ending up in the final binary because some linkers like LLD don't currently merge or strip these EH sections (see discussion [here](https://reviews.llvm.org/D83655)). This can bloat the ELF headers and string table significantly in binaries that contain many functions.
The new option is analogous to Clang's `-fno-unique-section-names`, and instructs LLVM to generate the same `.text` and `.gcc_except_table` section for each function, resulting in a smaller final binary.
The motivation to add this new option was because we have a binary that ended up with so many ELF sections (over 65,000) that it broke some existing ELF tools, which couldn't handle so many sections.
Here's our old binary:
```
$ readelf --sections old.elf | head -1
There are 71746 section headers, starting at offset 0x2a246508:
$ readelf --sections old.elf | grep shstrtab
[71742] .shstrtab STRTAB 0000000000000000 2977204c ad44bb 00 0 0 1
```
That's an 11MB+ string table. Here's the new binary using this option:
```
$ readelf --sections new.elf | head -1
There are 43 section headers, starting at offset 0x29143ca8:
$ readelf --sections new.elf | grep shstrtab
[40] .shstrtab STRTAB 0000000000000000 29143acc 0001db 00 0 0 1
```
The whole binary size went down by over 20MB, which is quite significant.
Implement -Z location-detail flag
This PR implements the `-Z location-detail` flag as described in https://github.com/rust-lang/rfcs/pull/2091 .
`-Z location-detail=val` controls what location details are tracked when using `caller_location`. This allows users to control what location details are printed as part of panic messages, by allowing them to exclude any combination of filenames, line numbers, and column numbers. This option is intended to provide users with a way to mitigate the size impact of `#[track_caller]`.
Some measurements of the savings of this approach on an embedded binary can be found here: https://github.com/rust-lang/rust/issues/70579#issuecomment-942556822 .
Closes#70580 (unless people want to leave that open as a place for discussion of further improvements).
This is my first real PR to rust, so any help correcting mistakes / understanding side effects / improving my tests is appreciated :)
I have one question: RFC 2091 specified this as a debugging option (I think that is what -Z implies?). Does that mean this can never be stabilized without a separate MCP? If so, do I need to submit an MCP now, or is the initial RFC specifying this option sufficient for this to be merged as is, and then an MCP would be needed for eventual stabilization?
Report fatal lexer errors in `--cfg` command line arguments
Fixes#89358. The erroneous behavior was apparently introduced by `@Mark-Simulacrum` in a678e31911; the idea is to silence individual parser errors and instead emit one catch-all error message after parsing. However, for the example in #89358, a fatal lexer error is created here:
edebf77e00/compiler/rustc_parse/src/lexer/mod.rs (L340-L349)
This fatal error aborts the compilation, and so the call to `new_parser_from_source_str()` never returns and the catch-all error message is never emitted. I have therefore changed the `SilentEmitter` to silence only non-fatal errors; with my changes, for the rustc invocation described in #89358:
```sh
rustc --cfg "abc\""
```
I get the following output:
```
error[E0765]: unterminated double quote string
|
= note: this error occurred on the command line: `--cfg=abc"`
```
This change adds a new compiler flag that can help reduce the size of
ELF binaries that contain many functions.
By default, when enabling function sections (which is the default for most
targets), the LLVM backend will generate different section names for each
function. For example, a function "func" would generate a section called
".text.func". Normally this is fine because the linker will merge all those
sections into a single one in the binary. However, starting with LLVM 12
(llvm/llvm-project@ee5d1a0), the backend will
also generate unique section names for exception handling, resulting in
thousands of ".gcc_except_table.*" sections ending up in the final binary
because some linkers don't currently merge or strip these EH sections.
This can bloat the ELF headers and string table significantly in
binaries that contain many functions.
The new option is analogous to Clang's -fno-unique-section-names, and
instructs LLVM to generate the same ".text" and ".gcc_except_table"
section for each function, resulting in smaller object files and
potentially a smaller final binary.
This largely involves implementing the options debug-info-for-profiling
and profile-sample-use and forwarding them on to LLVM.
AutoFDO can be used on x86-64 Linux like this:
rustc -O -Cdebug-info-for-profiling main.rs -o main
perf record -b ./main
create_llvm_prof --binary=main --out=code.prof
rustc -O -Cprofile-sample-use=code.prof main.rs -o main2
Now `main2` will have feedback directed optimization applied to it.
The create_llvm_prof tool can be obtained from this github repository:
https://github.com/google/autofdoFixes#64892.
Fix clippy lints
I'm currently working on allowing clippy to run on librustdoc after a discussion I had with `@Mark-Simulacrum.` So in the meantime, I fixed a few lints on the compiler crates.
Rework HIR API to make invocations of the hir_crate query harder.
`hir_crate` forces the recomputation of queries that depend on it.
This PR aims at avoiding useless invocations of `hir_crate` by making dependent code go through `tcx.hir()`.
Pass real crate-level attributes to `pre_expansion_lint`
The PR concerns the unstable feature `register_tool` (#66079).
The feature's implementation requires the attributes of the crate being compiled, so that when attributes like `allow(foo::bar)` are encountered, it can be verified that `register_tool(foo)` appears in the crate root.
However, the crate's attributes are not readily available during early lint passes. Specifically, on this line, `krate.attrs` appears to be the attributes of the current source file, not the attributes of the whole crate: bf642323d6/compiler/rustc_lint/src/context.rs (L815)
Consequently, "unknown tool" errors were being produced when `allow(foo::bar)` appeared in a submodule, even though `register_tool(foo)` appeared in the crate root.
EDITED: The proposed fix is to obtain the real crate-level attributes in `configure_and_expand` and pass them to `pre_expansion_lint`. (See `@petrochenkov's` [comment](https://github.com/rust-lang/rust/pull/89214#issuecomment-926927072) below.)
The original "prosed fix" text follows.
---
The proposed fix is to add an `error_on_unknown_tool` flag to `LintLevelsBuilder`. The flag controls whether "unknown tool" errors are emitted. The flag is set during late passes, but not earlier.
More specifically, this PR contains two commits:
* The first adds a `known-tool-in-submodule` UI test that does not currently pass.
* The second adds the `error_on_unknown_tool` flag. The new test passes with the addition of this flag.
This change has the added benefit of eliminating some errors that were duplicated in existing tests.
To the reviewer: please check that I implemented the UI test correctly.
This just applies the suggested fixes from the compatibility warnings,
leaving any that are in practice spurious in. This is primarily intended to
provide a starting point to identify possible fixes to the migrations (e.g., by
avoiding spurious warnings).
A secondary commit cleans these up where they are false positives (as is true in
many of the cases).
Gather module items after lowering.
This avoids having a non-local analysis inside lowering.
By implementing `hir_module_items` using a visitor, we make sure that iterations and visitors are consistent.
Introduce -Z remap-cwd-prefix switch
This switch remaps any absolute paths rooted under the current
working directory to a new value. This includes remapping the
debug info in `DW_AT_comp_dir` and `DW_AT_decl_file`.
Importantly, this flag does not require passing the current working
directory to the compiler, such that the command line can be
run on any machine (with the same input files) and produce the
same results. This is critical property for debugging compiler
issues that crop up on remote machines.
This is based on adetaylor's dbc4ae7cba
Major Change Proposal: https://github.com/rust-lang/compiler-team/issues/450
Discussed on #38322. Would resolve issue #87325.
Add -Z panic-in-drop={unwind,abort} command-line option
This PR changes `Drop` to abort if an unwinding panic attempts to escape it, making the process abort instead. This has several benefits:
- The current behavior when unwinding out of `Drop` is very unintuitive and easy to miss: unwinding continues, but the remaining drops in scope are simply leaked.
- A lot of unsafe code doesn't expect drops to unwind, which can lead to unsoundness:
- https://github.com/servo/rust-smallvec/issues/14
- https://github.com/bluss/arrayvec/issues/3
- There is a code size and compilation time cost to this: LLVM needs to generate extra landing pads out of all calls in a drop implementation. This can compound when functions are inlined since unwinding will then continue on to process drops in the callee, which can itself unwind, etc.
- Initial measurements show a 3% size reduction and up to 10% compilation time reduction on some crates (`syn`).
One thing to note about `-Z panic-in-drop=abort` is that *all* crates must be built with this option for it to be sound since it makes the compiler assume that dropping `Box<dyn Any>` will never unwind.
cc https://github.com/rust-lang/lang-team/issues/97
Avoid invoking the hir_crate query to traverse the HIR
Walking the HIR tree is done using the `hir_crate` query. However, this is unnecessary, since `hir_owner(CRATE_DEF_ID)` provides the same information. Since depending on `hir_crate` forces dependents to always be executed, this leads to unnecessary work.
By splitting HIR and attributes visits, we can avoid an edge to `hir_crate` when trying to visit the HIR tree.
Detect bare blocks with type ascription that were meant to be a `struct` literal
Address part of #34255.
Potential improvement: silence the other knock down errors in `issue-34255-1.rs`.
Instead of updating global state to mark attributes as used,
we now explicitly emit a warning when an attribute is used in
an unsupported position. As a side effect, we are to emit more
detailed warning messages (instead of just a generic "unused" message).
`Session.check_name` is removed, since its only purpose was to mark
the attribute as used. All of the callers are modified to use
`Attribute.has_name`
Additionally, `AttributeType::AssumedUsed` is removed - an 'assumed
used' attribute is implemented by simply not performing any checks
in `CheckAttrVisitor` for a particular attribute.
We no longer emit unused attribute warnings for the `#[rustc_dummy]`
attribute - it's an internal attribute used for tests, so it doesn't
mark sense to treat it as 'unused'.
With this commit, a large source of global untracked state is removed.
This was removed by #85284 in favor of -Zprofiler-runtime=<name>.
However the suggested -Zprofiler-runtime=None doesn't work because
"None" is treated as a crate name.
Since RFC 3052 soft deprecated the authors field anyway, hiding it from
crates.io, docs.rs, and making Cargo not add it by default, and it is
not generally up to date/useful information, we should remove it from
crates in this repo.
Add flag to configure `large_assignments` lint
The `large_assignments` lints detects moves over specified limit. The
limit is configured through `move_size_limit = "N"` attribute placed at
the root of a crate. When attribute is absent, the lint is disabled.
Make it possible to enable the lint without making any changes to the
source code, through a new flag `-Zmove-size-limit=N`. For example, to
detect moves exceeding 1023 bytes in a cargo crate, including all
dependencies one could use:
```
$ env RUSTFLAGS=-Zmove-size-limit=1024 cargo build -vv
```
Lint tracking issue #83518.
Add -Zfuture-incompat-test to assist with testing future-incompat reports.
This adds a `-Zfuture-incompat-test` cli flag to assist with testing future-incompatible reports. This flag causes all lints to be treated as a future-incompatible lint, and will emit a report for them. This is being added so that Cargo's testsuite can reliably test the reporting infrastructure. Right now, Cargo relies on using array_into_iter as a test subject. Since the breaking "future incompatible" lints are never intended to last forever, this means Cargo's testsuite would always need to keep changing to choose different lints (for example, #86330 proposed dropping that moniker for array_into_iter). With this flag, Cargo's tests can trigger any lint and check for the report.
The `large_assignments` lints detects moves over specified limit. The
limit is configured through `move_size_limit = "N"` attribute placed at
the root of a crate. When attribute is absent, the lint is disabled.
Make it possible to enable the lint without making any changes to the
source code, through a new flag `-Zmove-size-limit=N`. For example, to
detect moves exceeding 1023 bytes in a cargo crate, including all
dependencies one could use:
```
$ env RUSTFLAGS=-Zmove-size-limit=1024 cargo build -vv
```
Remove unused dependencies from compiler crates
Various compiler crates have dependencies that they don't appear to use. I used some scripting to detect such dependencies, filtered them based on some manual review, and removed those that do indeed appear to be entirely unused.
This creates a CSV with name "closure_profile_XXXXX.csv", where the
variable part is the process id of the compiler.
To profile a cargo project you can run one of the following depending on
if you're compiling a library or a binary:
```
cargo +stage1 rustc --lib -- -Zprofile-closures
cargo +stage1 rustc --bin -- -Zprofile-closures
```
Allow loading of llvm plugins on nightly
Based on a discussion in #82734 / with `@wsmoses.`
Mainly moves [this](0149bc4e7e) behind a -Z flag, so it can only be used on nightly,
as requested by `@nagisa` in https://github.com/rust-lang/rust/issues/82734#issuecomment-835863940
This change allows loading of llvm plugins like Enzyme.
Right now it also requires a shared library LLVM build of rustc for symbol resolution.
```rust
// test.rs
extern { fn __enzyme_autodiff(_: usize, ...) -> f64; }
fn square(x : f64) -> f64 {
return x * x;
}
fn main() {
unsafe {
println!("Hello, world {} {}!", square(3.0), __enzyme_autodiff(square as usize, 3.0));
}
}
```
```
./rustc test.rs -Z llvm-plugins="./LLVMEnzyme-12.so" -C passes="enzyme"
./test
Hello, world 9 6!
```
I will try to figure out how to simplify the usage and get this into stable in a later iteration,
but having this on nightly will already help testing further steps.
Provide option for specifying the profiler runtime
Currently, if `-Zinstrument-coverage` is enabled, the target is linked
against the `library/profiler_builtins` crate (which pulls in LLVM's
compiler-rt runtime).
This option enables backends to specify an alternative runtime crate for
handling injected instrumentation calls.
Don't use a generator for BoxedResolver
The generator is non-trivial and requires unsafe code anyway. Using regular unsafe code without a generator is much easier to follow.
Based on #85810 as it touches rustc_interface too.
Driver improvements
This PR contains a couple of cleanups for the driver and a few small improvements for the custom codegen backend interface. It also implements `--version` and `-Cpasses=list` support for custom codegen backends.
rustc: Store metadata-in-rlibs in object files
This commit updates how rustc compiler metadata is stored in rlibs.
Previously metadata was stored as a raw file that has the same format as
`--emit metadata`. After this commit, however, the metadata is encoded
into a small object file which has one section which is the contents of
the metadata.
The motivation for this commit is to fix a common case where #83730
arises. The problem is that when rustc crates a `dylib` crate type it
needs to include entire rlib files into the dylib, so it passes
`--whole-archive` (or the equivalent) to the linker. The problem with
this, though, is that the linker will attempt to read all files in the
archive. If the metadata file were left as-is (today) then the linker
would generate an error saying it can't read the file. The previous
solution was to alter the rlib just before linking, creating a new
archive in a temporary directory which has the metadata file removed.
This problem from before this commit is now removed if the metadata file
is stored in an object file that the linker can read. The only caveat we
have to take care of is to ensure that the linker never actually
includes the contents of the object file into the final output. We apply
similar tricks as the `.llvmbc` bytecode sections to do this.
This involved changing the metadata loading code a bit, namely updating
some of the LLVM C APIs used to use non-deprecated ones and fiddling
with the lifetimes a bit to get everything to work out. Otherwise though
this isn't intended to be a functional change really, only that metadata
is stored differently in archives now.
This should end up fixing #83730 because by default dylibs will no
longer have their rlib dependencies "altered" meaning that
split-debuginfo will continue to have valid paths pointing at the
original rlibs. (note that we still "alter" rlibs if LTO is enabled to
remove Rust object files and we also "alter" for the #[link(cfg)]
feature, but that's rarely used).
Closes#83730
This commit updates how rustc compiler metadata is stored in rlibs.
Previously metadata was stored as a raw file that has the same format as
`--emit metadata`. After this commit, however, the metadata is encoded
into a small object file which has one section which is the contents of
the metadata.
The motivation for this commit is to fix a common case where #83730
arises. The problem is that when rustc crates a `dylib` crate type it
needs to include entire rlib files into the dylib, so it passes
`--whole-archive` (or the equivalent) to the linker. The problem with
this, though, is that the linker will attempt to read all files in the
archive. If the metadata file were left as-is (today) then the linker
would generate an error saying it can't read the file. The previous
solution was to alter the rlib just before linking, creating a new
archive in a temporary directory which has the metadata file removed.
This problem from before this commit is now removed if the metadata file
is stored in an object file that the linker can read. The only caveat we
have to take care of is to ensure that the linker never actually
includes the contents of the object file into the final output. We apply
similar tricks as the `.llvmbc` bytecode sections to do this.
This involved changing the metadata loading code a bit, namely updating
some of the LLVM C APIs used to use non-deprecated ones and fiddling
with the lifetimes a bit to get everything to work out. Otherwise though
this isn't intended to be a functional change really, only that metadata
is stored differently in archives now.
This should end up fixing #83730 because by default dylibs will no
longer have their rlib dependencies "altered" meaning that
split-debuginfo will continue to have valid paths pointing at the
original rlibs. (note that we still "alter" rlibs if LTO is enabled to
remove Rust object files and we also "alter" for the #[link(cfg)]
feature, but that's rarely used).
Closes#83730
Remove unused feature gates
The first commit removes a usage of a feature gate, but I don't expect it to be controversial as the feature gate was only used to workaround a limitation of rust in the past. (closures never being `Clone`)
The second commit uses `#[allow_internal_unstable]` to avoid leaking the `trusted_step` feature gate usage from inside the index newtype macro. It didn't work for the `min_specialization` feature gate though.
The third commit removes (almost) all feature gates from the compiler that weren't used anyway.
Reduce the amount of untracked state in TyCtxt
Access to untracked global state may generate instances of #84970.
The GlobalCtxt contains the lowered HIR, the resolver outputs and interners.
By wrapping the resolver inside a query, we make sure those accesses are properly tracked.
As a no_hash query, all dependent queries essentially become `eval_always`,
what they should have been from the beginning.
Don't sort a `Vec` before computing its `DepTrackingHash`
Previously, we sorted the vec prior to hashing, making the hash
independent of the original (command-line argument) order. However, the
original vec was still always kept in the original order, so we were
relying on the rest of the compiler always working with it in an
'order-independent' way.
This assumption was not being upheld by the `native_libraries` query -
the order of the entires in its result depends on the order of entries
in `Options.libs`. This lead to an 'unstable fingerprint' ICE when the
`-l` arguments were re-ordered.
This PR removes the sorting logic entirely. Re-ordering command-line
arguments (without adding/removing/changing any arguments) seems like a
really niche use case, and correctly optimizing for it would require
additional work. By always hashing arguments in their original order, we
can entirely avoid a cause of 'unstable fingerprint' errors.
Previously, we sorted the vec prior to hashing, making the hash
independent of the original (command-line argument) order. However, the
original vec was still always kept in the original order, so we were
relying on the rest of the compiler always working with it in an
'order-independent' way.
This assumption was not being upheld by the `native_libraries` query -
the order of the entires in its result depends on the order of entries
in `Options.libs`. This lead to an 'unstable fingerprint' ICE when the
`-l` arguments were re-ordered.
This PR removes the sorting logic entirely. Re-ordering command-line
arguments (without adding/removing/changing any arguments) seems like a
really niche use case, and correctly optimizing for it would require
additional work. By always hashing arguments in their original order, we
can entirely avoid a cause of 'unstable fingerprint' errors.
Currently, if `-Zinstrument-coverage` is enabled, the target is linked
against the `library/profiler_builtins` crate (which pulls in LLVM's
compiler-rt runtime).
This option enables backends to specify an alternative runtime crate for
handling injected instrumentation calls.
Remove CrateNum parameter for queries that only work on local crate
The pervasive `CrateNum` parameter is a remnant of the multi-crate rustc idea.
Using `()` as query key in those cases avoids having to worry about the validity of the query key.
Introduce the beginning of a THIR unsafety checker
This poses the foundations for the THIR unsafety checker, so that it can be implemented incrementally:
- implements a rudimentary `Visitor` for the THIR (which will definitely need some tweaking in the future)
- introduces a new `-Zthir-unsafeck` flag which tells the compiler to use THIR unsafeck instead of MIR unsafeck
- implements detection of unsafe functions
- adds revisions to the UI tests to test THIR unsafeck alongside MIR unsafeck
This uses a very simple query design, where bodies are unsafety-checked on a body per body basis. This however has some big flaws:
- the unsafety-checker builds the THIR itself, which means a lot of work is duplicated with MIR building constructing its own copy of the THIR
- unsafety-checking closures is currently completely wrong: closures should take into account the "safety context" in which they are created, here we are considering that closures are always a safe context
I had intended to fix these problems in follow-up PRs since they are always gated under the `-Zthir-unsafeck` flag (which is explicitely noted to be unsound).
r? `@nikomatsakis`
cc https://github.com/rust-lang/project-thir-unsafeck/issues/3https://github.com/rust-lang/project-thir-unsafeck/issues/7
Fix `--remap-path-prefix` not correctly remapping `rust-src` component paths and unify handling of path mapping with virtualized paths
This PR fixes#73167 ("Binaries end up containing path to the rust-src component despite `--remap-path-prefix`") by preventing real local filesystem paths from reaching compilation output if the path is supposed to be remapped.
`RealFileName::Named` introduced in #72767 is now renamed as `LocalPath`, because this variant wraps a (most likely) valid local filesystem path.
`RealFileName::Devirtualized` is renamed as `Remapped` to be used for remapped path from a real path via `--remap-path-prefix` argument, as well as real path inferred from a virtualized (during compiler bootstrapping) `/rustc/...` path. The `local_path` field is now an `Option<PathBuf>`, as it will be set to `None` before serialisation, so it never reaches any build output. Attempting to serialise a non-`None` `local_path` will cause an assertion faliure.
When a path is remapped, a `RealFileName::Remapped` variant is created. The original path is preserved in `local_path` field and the remapped path is saved in `virtual_name` field. Previously, the `local_path` is directly modified which goes against its purpose of "suitable for reading from the file system on the local host".
`rustc_span::SourceFile`'s fields `unmapped_path` (introduced by #44940) and `name_was_remapped` (introduced by #41508 when `--remap-path-prefix` feature originally added) are removed, as these two pieces of information can be inferred from the `name` field: if it's anything other than a `FileName::Real(_)`, or if it is a `FileName::Real(RealFileName::LocalPath(_))`, then clearly `name_was_remapped` would've been false and `unmapped_path` would've been `None`. If it is a `FileName::Real(RealFileName::Remapped{local_path, virtual_name})`, then `name_was_remapped` would've been true and `unmapped_path` would've been `Some(local_path)`.
cc `@eddyb` who implemented `/rustc/...` path devirtualisation
This commit implements both the native linking modifiers infrastructure
as well as an initial attempt at the individual modifiers from the RFC.
It also introduces a feature flag for the general syntax along with
individual feature flags for each modifier.
Implement RFC 1260 with feature_name `imported_main`.
This is the second extraction part of #84062 plus additional adjustments.
This (mostly) implements RFC 1260.
However there's still one test case failure in the extern crate case. Maybe `LocalDefId` doesn't work here? I'm not sure.
cc https://github.com/rust-lang/rust/issues/28937
r? `@petrochenkov`
Add TRACKED_NO_CRATE_HASH and use it for `--remap-path-prefix`
I verified locally that this fixes https://github.com/rust-lang/rust/issues/66955.
r? `@Aaron1011` (feel free to reassign)
This is necessary for options that should invalidate the incremental
hash but *not* affect the crate hash (e.g. --remap-path-prefix).
This doesn't add `for_crate_hash` to the trait directly because it's not
relevant for *types*, only for *options*, which are fields on a larger
struct. Instead, it adds a new `SUBSTRUCT` directive for options, which
does take a `for_crate_hash` parameter.
- Use TRACKED_NO_CRATE_HASH for --remap-path-prefix
- Add test that `remap_path_prefix` is tracked
- Reduce duplication in the test suite to avoid future churn
Find codegen backends in more locations
* Search in the sysroot passed using `--sysroot` in addition to the default sysroot.
* Search for `librustc_codegen_$name.so` in addition to `librustc_codegen_$name-$release.so`.
This combined would allow putting `librustc_codegen_cranelift.so` in the right location of a sysroot passed using `--sysroot`.
Add an unstable --json=unused-externs flag to print unused externs
This adds an unstable flag to print a list of the extern names not used by cargo.
This PR will enable cargo to collect unused dependencies from all units and provide warnings.
The companion PR to cargo is: https://github.com/rust-lang/cargo/pull/8437
The goal is eventual stabilization of this flag in rustc as well as in cargo.
Discussion of this feature is mostly contained inside these threads: #57274#72342#72603
The feature builds upon the internal datastructures added by #72342
Externs are uniquely identified by name and the information is sufficient for cargo.
If the mode is enabled, rustc will print json messages like:
```
{"unused_extern_names":["byteorder","openssl","webpki"]}
```
For a crate that got passed byteorder, openssl and webpki dependencies but needed none of them.
### Q: Why not pass -Wunused-crate-dependencies?
A: See [ehuss's comment here](https://github.com/rust-lang/rust/issues/57274#issuecomment-624839355)
TLDR: it's cleaner. Rust's warning system wasn't built to be filtered or edited by cargo.
Even a basic implementation of the feature would have to change the "n warnings emitted" line that rustc prints at the end.
Cargo ideally wants to synthesize its own warnings anyways. For example, it would be hard for rustc to emit warnings like
"dependency foo is only used by dev targets", suggesting to make it a dev-dependency instead.
### Q: Make rustc emit used or unused externs?
A: Emitting used externs has the advantage that it simplifies cargo's collection job.
However, emitting unused externs creates less data to be communicated between rustc and cargo.
Often you want to paste a cargo command obtained from `cargo build -vv` for doing something
completely unrelated. The message is emitted always, even if no warning or error is emitted.
At that point, even this tiny difference in "noise" matters. That's why I went with emitting unused externs.
### Q: One json msg per extern or a collective json msg?
A: Same as above, the data format should be concise. Having 30 lines for the 30 crates a crate uses would be disturbing to readers.
Also it helps the cargo implementation to know that there aren't more unused deps coming.
### Q: Why use names of externs instead of e.g. paths?
A: Names are both sufficient as well as neccessary to uniquely identify a passed `--extern` arg.
Names are sufficient because you *must* pass a name when passing an `--extern` arg.
Passing a path is optional on the other hand so rustc might also figure out a crate's location from the file system.
You can also put multiple paths for the same extern name, via e.g. `--extern hello=/usr/lib/hello.rmeta --extern hello=/usr/local/lib/hello.rmeta`,
but rustc will only ever use one of those paths.
Also, paths don't identify a dependency uniquely as it is possible to have multiple different extern names point to the same path.
So paths are ill-suited for identification.
### Q: What about 2015 edition crates?
A: They are fully supported.
Even on the 2015 edition, an explicit `--extern` flag is is required to enable `extern crate foo;` to work (outside of sysroot crates, which this flag doesn't warn about anyways).
So the lint would still fire on 2015 edition crates if you haven't included a dependency specified in Cargo.toml using `extern crate foo;` or similar.
The lint won't fire if your sole use in the crate is through a `extern crate foo;` statement, but that's not its job.
For detecting unused `extern crate foo` statements, there is the `unused_extern_crates` lint
which can be enabled by `#![warn(unused_extern_crates)]` or similar.
cc ```@jsgf``` ```@ehuss``` ```@petrochenkov``` ```@estebank```
Don't duplicate the extern providers once for each crate
This should give a small perf improvement for small crates by avoiding a memcpy of a pretty big struct for each loaded crate. In addition would be useful for replacing the sequential `CrateNum` everywhere with the hash based `StableCrateId` introduced in #81635, which would allow avoiding remapping of `CrateNum`'s when loading crate metadata. While this PR is not strictly needed for that, it is necessary to prevent a performance loss due to it.
I think this duplication was done in https://github.com/rust-lang/rust/pull/40008 (which introduced the query system) to make it possible to compile multiple crates in a single session in the future. I think this is unlikely to be implemented any time soon. In addition this PR can easily be reverted if necessary to implement this.
- Add back `HirIdVec`, with a comment that it will soon be used.
- Add back `*_region` functions, with a comment they may soon be used.
- Remove `-Z borrowck_stats` completely. It didn't do anything.
- Remove `make_nop` completely.
- Add back `current_loc`, which is used by an out-of-tree tool.
- Fix style nits
- Remove `AtomicCell` with `cfg(parallel_compiler)` for consistency.
coverage bug fixes and optimization support
Adjusted LLVM codegen for code compiled with `-Zinstrument-coverage` to
address multiple, somewhat related issues.
Fixed a significant flaw in prior coverage solution: Every counter
generated a new counter variable, but there should have only been one
counter variable per function. This appears to have bloated .profraw
files significantly. (For a small program, it increased the size by
about 40%. I have not tested large programs, but there is anecdotal
evidence that profraw files were way too large. This is a good fix,
regardless, but hopefully it also addresses related issues.
Fixes: #82144
Invalid LLVM coverage data produced when compiled with -C opt-level=1
Existing tests now work up to at least `opt-level=3`. This required a
detailed analysis of the LLVM IR, comparisons with Clang C++ LLVM IR
when compiled with coverage, and a lot of trial and error with codegen
adjustments.
The biggest hurdle was figuring out how to continue to support coverage
results for unused functions and generics. Rust's coverage results have
three advantages over Clang's coverage results:
1. Rust's coverage map does not include any overlapping code regions,
making coverage counting unambiguous.
2. Rust generates coverage results (showing zero counts) for all unused
functions, including generics. (Clang does not generate coverage for
uninstantiated template functions.)
3. Rust's unused functions produce minimal stubbed functions in LLVM IR,
sufficient for including in the coverage results; while Clang must
generate the complete LLVM IR for each unused function, even though
it will never be called.
This PR removes the previous hack of attempting to inject coverage into
some other existing function instance, and generates dedicated instances
for each unused function. This change, and a few other adjustments
(similar to what is required for `-C link-dead-code`, but with lower
impact), makes it possible to support LLVM optimizations.
Fixes: #79651
Coverage report: "Unexecuted instantiation:..." for a generic function
from multiple crates
Fixed by removing the aforementioned hack. Some "Unexecuted
instantiation" notices are unavoidable, as explained in the
`used_crate.rs` test, but `-Zinstrument-coverage` has new options to
back off support for either unused generics, or all unused functions,
which avoids the notice, at the cost of less coverage of unused
functions.
Fixes: #82875
Invalid LLVM coverage data produced with crate brotli_decompressor
Fixed by disabling the LLVM function attribute that forces inlining, if
`-Z instrument-coverage` is enabled. This attribute is applied to
Rust functions with `#[inline(always)], and in some cases, the forced
inlining breaks coverage instrumentation and reports.
FYI: `@wesleywiser`
r? `@tmandry`
Adjusted LLVM codegen for code compiled with `-Zinstrument-coverage` to
address multiple, somewhat related issues.
Fixed a significant flaw in prior coverage solution: Every counter
generated a new counter variable, but there should have only been one
counter variable per function. This appears to have bloated .profraw
files significantly. (For a small program, it increased the size by
about 40%. I have not tested large programs, but there is anecdotal
evidence that profraw files were way too large. This is a good fix,
regardless, but hopefully it also addresses related issues.
Fixes: #82144
Invalid LLVM coverage data produced when compiled with -C opt-level=1
Existing tests now work up to at least `opt-level=3`. This required a
detailed analysis of the LLVM IR, comparisons with Clang C++ LLVM IR
when compiled with coverage, and a lot of trial and error with codegen
adjustments.
The biggest hurdle was figuring out how to continue to support coverage
results for unused functions and generics. Rust's coverage results have
three advantages over Clang's coverage results:
1. Rust's coverage map does not include any overlapping code regions,
making coverage counting unambiguous.
2. Rust generates coverage results (showing zero counts) for all unused
functions, including generics. (Clang does not generate coverage for
uninstantiated template functions.)
3. Rust's unused functions produce minimal stubbed functions in LLVM IR,
sufficient for including in the coverage results; while Clang must
generate the complete LLVM IR for each unused function, even though
it will never be called.
This PR removes the previous hack of attempting to inject coverage into
some other existing function instance, and generates dedicated instances
for each unused function. This change, and a few other adjustments
(similar to what is required for `-C link-dead-code`, but with lower
impact), makes it possible to support LLVM optimizations.
Fixes: #79651
Coverage report: "Unexecuted instantiation:..." for a generic function
from multiple crates
Fixed by removing the aforementioned hack. Some "Unexecuted
instantiation" notices are unavoidable, as explained in the
`used_crate.rs` test, but `-Zinstrument-coverage` has new options to
back off support for either unused generics, or all unused functions,
which avoids the notice, at the cost of less coverage of unused
functions.
Fixes: #82875
Invalid LLVM coverage data produced with crate brotli_decompressor
Fixed by disabling the LLVM function attribute that forces inlining, if
`-Z instrument-coverage` is enabled. This attribute is applied to
Rust functions with `#[inline(always)], and in some cases, the forced
inlining breaks coverage instrumentation and reports.
Update to rustc-rayon 0.3.1
This pulls in rust-lang/rustc-rayon#8 to fix#81425. (h/t `@ammaraskar)`
That revealed weak constraints on `rustc_arena::DropArena`, because its
`DropType` was holding type-erased raw pointers to generic `T`. We can
implement `Send` for `DropType` (under `cfg(parallel_compiler)`) by
requiring all `T: Send` before they're type-erased.
This pulls in rust-lang/rustc-rayon#8 to fix#81425. (h/t @ammaraskar)
That revealed weak constraints on `rustc_arena::DropArena`, because its
`DropType` was holding type-erased raw pointers to generic `T`. We can
implement `Send` for `DropType` (under `cfg(parallel_compiler)`) by
requiring all `T: Send` before they're type-erased.
Remove the -Zinsert-sideeffect
This removes all of the code we had in place to work-around LLVM's
handling of forward progress. From this removal excluded is a workaround
where we'd insert a `sideeffect` into clearly infinite loops such as
`loop {}`. This code remains conditionally effective when the LLVM
version is earlier than 12.0, which fixed the forward progress related
miscompilations at their root.
This removes all of the code we had in place to work-around LLVM's
handling of forward progress. From this removal excluded is a workaround
where we'd insert a `sideeffect` into clearly infinite loops such as
`loop {}`. This code remains conditionally effective when the LLVM
version is earlier than 12.0, which fixed the forward progress related
miscompilations at their root.
expand: Refactor module loading
This is an accompanying PR to https://github.com/rust-lang/rust/pull/82399, but they can be landed independently.
See individual commits for more details.
Anyone should be able to review this equally well because all people actually familiar with this code left the project.
bypass auto_da_alloc for metadata files
This saves about 0.7% when rerunning the UI test suite. I.e. when the metadata files exist and will be overwritten. No improvements expected for a clean build. So it might show up in incr-patched perf results.
```
regular rename:
Benchmark #1: touch src/tools/compiletest/src/main.rs ; RUSTC_WRAPPER="" schedtool -B -e ./x.py test src/test/ui
Time (mean ± σ): 47.305 s ± 0.170 s [User: 1631.540 s, System: 412.648 s]
Range (min … max): 47.125 s … 47.856 s 20 runs
non-durable rename:
Benchmark #1: touch src/tools/compiletest/src/main.rs ; RUSTC_WRAPPER="" schedtool -B -e ./x.py test src/test/ui
Time (mean ± σ): 46.930 s ± 0.064 s [User: 1634.344 s, System: 396.038 s]
Range (min … max): 46.759 s … 47.043 s 20 runs
```
There are more places that trigger auto_da_alloc behavior by overwriting existing files with O_TRUNC, but those are much harder to locate because `O_TRUNC` is set on `open()` but the writeback is triggered on `close()`. The latter is the part which shows up in profiles.
Cleanup `PpMode` and friends
This PR:
- Separates `PpSourceMode` and `PpHirMode` to remove invalid states
- Renames the variant to remove the redundant `Ppm` prefix
- Adds basic documentation for the different pretty-print modes
- Cleanups some code to make it more idiomatic
Not sure if this is actually useful, but it looks cleaner to me.
Make `treat_err_as_bug` Option<NonZeroUsize>
`rustc -Z treat-err-as-bug=N` already requires `N` to be nonzero when the argument is parsed, so changing the type from `Option<usize>` to `Option<NonZeroUsize>` is a low-hanging fruit in terms of layout optimization.
* Search in the sysroot passed using `--sysroot` in addition to the
default sysroot.
* Search for `librustc_codegen_$name.so` in addition to
`librustc_codegen_$name-$release.so`.
cfg(version): treat nightlies as complete
This PR makes cfg(version) treat the nightlies
for version 1.n.0 as 1.n.0, even though that nightly
version might not have all stabilizations and features
of the released 1.n.0. This is done for greater
convenience for people who want to test a newly
stabilized feature on nightly, or in other words,
give newly stabilized features as many eyeballs
as possible.
For users who wish to pin nightlies, this commit adds
a -Z assume-incomplete-release option that they can
enable if they run into any issues due to this change.
Implements the suggestion in https://github.com/rust-lang/rust/issues/64796#issuecomment-640851454
This commit makes cfg(version) treat the nightlies
for version 1.n.0 as 1.n.0, even though that nightly
version might not have all stabilizations and features
of the released 1.n.0. This is done for greater
convenience for people who want to test a newly
stabilized feature on nightly.
For users who wish to pin nightlies, this commit adds
a -Z assume-incomplete-release option that they can
enable if there are any issues due to this change.
rustc: Stabilize `-Zrun-dsymutil` as `-Csplit-debuginfo`
This commit adds a new stable codegen option to rustc,
`-Csplit-debuginfo`. The old `-Zrun-dsymutil` flag is deleted and now
subsumed by this stable flag. Additionally `-Zsplit-dwarf` is also
subsumed by this flag but still requires `-Zunstable-options` to
actually activate. The `-Csplit-debuginfo` flag takes one of
three values:
* `off` - This indicates that split-debuginfo from the final artifact is
not desired. This is not supported on Windows and is the default on
Unix platforms except macOS. On macOS this means that `dsymutil` is
not executed.
* `packed` - This means that debuginfo is desired in one location
separate from the main executable. This is the default on Windows
(`*.pdb`) and macOS (`*.dSYM`). On other Unix platforms this subsumes
`-Zsplit-dwarf=single` and produces a `*.dwp` file.
* `unpacked` - This means that debuginfo will be roughly equivalent to
object files, meaning that it's throughout the build directory
rather than in one location (often the fastest for local development).
This is not the default on any platform and is not supported on Windows.
Each target can indicate its own default preference for how debuginfo is
handled. Almost all platforms default to `off` except for Windows and
macOS which default to `packed` for historical reasons.
Some equivalencies for previous unstable flags with the new flags are:
* `-Zrun-dsymutil=yes` -> `-Csplit-debuginfo=packed`
* `-Zrun-dsymutil=no` -> `-Csplit-debuginfo=unpacked`
* `-Zsplit-dwarf=single` -> `-Csplit-debuginfo=packed`
* `-Zsplit-dwarf=split` -> `-Csplit-debuginfo=unpacked`
Note that `-Csplit-debuginfo` still requires `-Zunstable-options` for
non-macOS platforms since split-dwarf support was *just* implemented in
rustc.
There's some more rationale listed on #79361, but the main gist of the
motivation for this commit is that `dsymutil` can take quite a long time
to execute in debug builds and provides little benefit. This means that
incremental compile times appear that much worse on macOS because the
compiler is constantly running `dsymutil` over every single binary it
produces during `cargo build` (even build scripts!). Ideally rustc would
switch to not running `dsymutil` by default, but that's a problem left
to get tackled another day.
Closes#79361
This commit adds a new stable codegen option to rustc,
`-Csplit-debuginfo`. The old `-Zrun-dsymutil` flag is deleted and now
subsumed by this stable flag. Additionally `-Zsplit-dwarf` is also
subsumed by this flag but still requires `-Zunstable-options` to
actually activate. The `-Csplit-debuginfo` flag takes one of
three values:
* `off` - This indicates that split-debuginfo from the final artifact is
not desired. This is not supported on Windows and is the default on
Unix platforms except macOS. On macOS this means that `dsymutil` is
not executed.
* `packed` - This means that debuginfo is desired in one location
separate from the main executable. This is the default on Windows
(`*.pdb`) and macOS (`*.dSYM`). On other Unix platforms this subsumes
`-Zsplit-dwarf=single` and produces a `*.dwp` file.
* `unpacked` - This means that debuginfo will be roughly equivalent to
object files, meaning that it's throughout the build directory
rather than in one location (often the fastest for local development).
This is not the default on any platform and is not supported on Windows.
Each target can indicate its own default preference for how debuginfo is
handled. Almost all platforms default to `off` except for Windows and
macOS which default to `packed` for historical reasons.
Some equivalencies for previous unstable flags with the new flags are:
* `-Zrun-dsymutil=yes` -> `-Csplit-debuginfo=packed`
* `-Zrun-dsymutil=no` -> `-Csplit-debuginfo=unpacked`
* `-Zsplit-dwarf=single` -> `-Csplit-debuginfo=packed`
* `-Zsplit-dwarf=split` -> `-Csplit-debuginfo=unpacked`
Note that `-Csplit-debuginfo` still requires `-Zunstable-options` for
non-macOS platforms since split-dwarf support was *just* implemented in
rustc.
There's some more rationale listed on #79361, but the main gist of the
motivation for this commit is that `dsymutil` can take quite a long time
to execute in debug builds and provides little benefit. This means that
incremental compile times appear that much worse on macOS because the
compiler is constantly running `dsymutil` over every single binary it
produces during `cargo build` (even build scripts!). Ideally rustc would
switch to not running `dsymutil` by default, but that's a problem left
to get tackled another day.
Closes#79361
