Simplify the internal API for declaring command-line options
The internal APIs for declaring command-line options are old, and intimidatingly complex. This PR replaces them with a single function that takes explicit `stability` and `kind` arguments, making it easier to see how each option is handled, and whether it is treated as stable or unstable.
We also don't appear to have any tests for the output of `rustc --help` and similar, so I've added a run-make test to verify that this PR doesn't change any output. (There is already a similar run-make test for rustdoc's help output.)
---
The librustdoc changes are simply adjusting to updated compiler APIs; no functional change intended.
---
A side-effect of these changes is that rustfmt can once again format the entirety of these option declaration lists, which it was not doing before.
Tweak detection of multiple crate versions to be more encompassing
Previously, we only emitted the additional context if the type was in the same crate as the trait that appeared multiple times in the dependency tree. Now, we look at all traits looking for two with the same name in different crates with the same crate number, and we are more flexible looking for the types involved. This will work even if the type that implements the wrong trait version is from a different crate entirely.
```
error[E0277]: the trait bound `CustomErrorHandler: ErrorHandler` is not satisfied because the trait comes from a different crate version
--> src/main.rs:5:17
|
5 | cnb_runtime(CustomErrorHandler {});
| ^^^^^^^^^^^^^^^^^^^^^ the trait `ErrorHandler` is not implemented for `CustomErrorHandler`
|
note: there are multiple different versions of crate `c` in the dependency graph
--> /home/gh-estebank/testcase-rustc-crate-version-mismatch/c-v0.2/src/lib.rs:1:1
|
1 | pub trait ErrorHandler {}
| ^^^^^^^^^^^^^^^^^^^^^^ this is the required trait
|
::: src/main.rs:1:5
|
1 | use b::CustomErrorHandler;
| - one version of crate `c` is used here, as a dependency of crate `b`
2 | use c::cnb_runtime;
| - one version of crate `c` is used here, as a direct dependency of the current crate
|
::: /home/gh-estebank/testcase-rustc-crate-version-mismatch/b/src/lib.rs:1:1
|
1 | pub struct CustomErrorHandler {}
| ----------------------------- this type doesn't implement the required trait
|
::: /home/gh-estebank/testcase-rustc-crate-version-mismatch/c-v0.1/src/lib.rs:1:1
|
1 | pub trait ErrorHandler {}
| ---------------------- this is the found trait
= note: two types coming from two different versions of the same crate are different types even if they look the same
= help: you can use `cargo tree` to explore your dependency tree
```
Fix#89143.
```
error[E0277]: the trait bound `dep_2_reexport::Type: Trait` is not satisfied because the trait comes from a different crate version
--> multiple-dep-versions.rs:7:18
|
7 | do_something(Type);
| ^^^^ the trait `Trait` is not implemented for `dep_2_reexport::Type`
|
note: there are multiple different versions of crate `dependency` in the dependency graph
--> /home/gh-estebank/rust/build/x86_64-unknown-linux-gnu/test/run-make/crate-loading/rmake_out/multiple-dep-versions-1.rs:4:1
|
3 | pub struct Type(pub i32);
| --------------- this type implements the required trait
4 | pub trait Trait {
| ^^^^^^^^^^^^^^^ this is the required trait
|
::: multiple-dep-versions.rs:1:1
|
1 | extern crate dep_2_reexport;
| ---------------------------- one version of crate `dependency` is used here, as a dependency of crate `foo`
2 | extern crate dependency;
| ------------------------ one version of crate `dependency` is used here, as a direct dependency of the current crate
|
::: /home/gh-estebank/rust/build/x86_64-unknown-linux-gnu/test/run-make/crate-loading/rmake_out/multiple-dep-versions-2.rs:3:1
|
3 | pub struct Type;
| --------------- this type doesn't implement the required trait
4 | pub trait Trait {
| --------------- this is the found trait
= note: two types coming from two different versions of the same crate are different types even if they look the same
= help: you can use `cargo tree` to explore your dependency tree
```
The approach to accomplish this is a HACK, and we'd want a better way to do this. I believe that moving E0277 to be a structured diagnostic would help in that regard.
```
error[E0277]: the trait bound `dep_2_reexport::Type: Trait` is not satisfied because the trait comes from a different crate version
--> multiple-dep-versions.rs:7:18
|
7 | do_something(Type);
| ^^^^ the trait `Trait` is not implemented for `dep_2_reexport::Type`
|
note: there are multiple different versions of crate `dependency` in the dependency graph
--> /home/gh-estebank/rust/build/x86_64-unknown-linux-gnu/test/run-make/crate-loading/rmake_out/multiple-dep-versions-1.rs:4:1
|
3 | pub struct Type(pub i32);
| --------------- this type implements the required trait
4 | pub trait Trait {
| ^^^^^^^^^^^^^^^ this is the required trait
|
::: multiple-dep-versions.rs:1:1
|
1 | extern crate dep_2_reexport;
| ---------------------------- one version of crate `dependency` is used here, as a dependency of crate `foo`
2 | extern crate dependency;
| ------------------------ one version of crate `dependency` is used here, as a direct dependency of the current crate
|
::: /home/gh-estebank/rust/build/x86_64-unknown-linux-gnu/test/run-make/crate-loading/rmake_out/multiple-dep-versions-2.rs:3:1
|
3 | pub struct Type;
| --------------- this type doesn't implement the required trait
4 | pub trait Trait {
| --------------- this is the found trait
= note: two types coming from two different versions of the same crate are different types even if they look the same
= help: you can use `cargo tree` to explore your dependency tree
note: required by a bound in `do_something`
--> /home/gh-estebank/rust/build/x86_64-unknown-linux-gnu/test/run-make/crate-loading/rmake_out/multiple-dep-versions-1.rs:12:24
|
12 | pub fn do_something<X: Trait>(_: X) {}
| ^^^^^ required by this bound in `do_something`
```
Previously, we only emitted the additional context if the type was in the same crate as the trait that appeared multiple times in the dependency tree. Now, we look at all traits looking for two with the same name in different crates with the same crate number, and we are more flexible looking for the types involved. This will work even if the type that implements the wrong trait version is from a different crate entirely.
```
error[E0277]: the trait bound `CustomErrorHandler: ErrorHandler` is not satisfied
--> src/main.rs:5:17
|
5 | cnb_runtime(CustomErrorHandler {});
| ----------- ^^^^^^^^^^^^^^^^^^^^^ the trait `ErrorHandler` is not implemented for `CustomErrorHandler`
| |
| required by a bound introduced by this call
|
help: you have multiple different versions of crate `c` in your dependency graph
--> src/main.rs:1:5
|
1 | use b::CustomErrorHandler;
| ^ one version of crate `c` is used here, as a dependency of crate `b`
2 | use c::cnb_runtime;
| ^ one version of crate `c` is used here, as a direct dependency of the current crate
note: two types coming from two different versions of the same crate are different types even if they look the same
--> /home/gh-estebank/testcase-rustc-crate-version-mismatch/c-v0.2/src/lib.rs:1:1
|
1 | pub trait ErrorHandler {}
| ^^^^^^^^^^^^^^^^^^^^^^ this is the required trait
|
::: /home/gh-estebank/testcase-rustc-crate-version-mismatch/b/src/lib.rs:1:1
|
1 | pub struct CustomErrorHandler {}
| ----------------------------- this type doesn't implement the required trait
|
::: /home/gh-estebank/testcase-rustc-crate-version-mismatch/c-v0.1/src/lib.rs:1:1
|
1 | pub trait ErrorHandler {}
| ---------------------- this is the found trait
= help: you can use `cargo tree` to explore your dependency tree
note: required by a bound in `cnb_runtime`
--> /home/gh-estebank/testcase-rustc-crate-version-mismatch/c-v0.2/src/lib.rs:3:41
|
3 | pub fn cnb_runtime(_error_handler: impl ErrorHandler) {}
| ^^^^^^^^^^^^ required by this bound in `cnb_runtime`
```
Fix#89143.
rustc_codegen_llvm: Add a new 'pc' option to branch-protection
Add a new 'pc' option to -Z branch-protection for aarch64 that enables the use of PC as a diversifier in PAC branch protection code.
When the pauth-lr target feature is enabled in combination with -Z branch-protection=pac-ret,pc, the new 9.5-a instructions (pacibsppc, retaasppc, etc) will be generated.
Add a new 'pc' option to -Z branch-protection for aarch64 that
enables the use of PC as a diversifier in PAC branch protection code.
When the pauth-lr target feature is enabled in combination
with -Z branch-protection=pac-ret,pc, the new 9.5-a instructions
(pacibsppc, retaasppc, etc) will be generated.
this makes it much easier to understand test failures.
before:
```
diff of stderr:
1 error: linking with `LINKER` failed: exit status: 1
2 |
- ld: Undefined symbols:
4 _CFRunLoopGetTypeID, referenced from:
5 clang: error: linker command failed with exit code 1 (use -v to see invocation)
```
after:
```
=== HAYSTACK ===
error: linking with `cc` failed: exit status: 1
|
= note: use `--verbose` to show all linker arguments
= note: Undefined symbols for architecture arm64:
"_CFRunLoopGetTypeID", referenced from:
main::main::hbb553f5dda62d3ea in main.main.d17f5fbe6225cf88-cgu.0.rcgu.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: aborting due to 1 previous error
=== NEEDLE ===
_CFRunLoopGetTypeID\.?, referenced from:
thread 'main' panicked at /Users/jyn/git/rust-lang/rust/tests/run-make/linkage-attr-framework/rmake.rs:22:10:
needle was not found in haystack
```
this also fixes a failure related to missing whitespace; we don't actually care about whitespace in this test.
Run the full stage 2 `run-make` test suite in `x86_64-gnu-debug`
Run the full `run-make` test suite in the `x86_64-gnu-debug` CI job. This is currently the *only* CI job where `//@ needs-force-clang-based-test` will be satisfied, so some `run-make` tests will literally never be run otherwise. Before this PR, the CI job only ran `run-make` tests which contains the substring `clang` in its test name, which is both (1) a footgun because it's very easy to forget and (2) it masks tests that would otherwise fail (even failing to compile) because the test is skipped if doesn't have a `clang` in its test name.
With the environment of `x86_64-gnu-debug`, two `run-make` tests failed before this PR:
1. `tests/run-make/issue-84395-lto-embed-bitcode/rmake.rs`: this was broken for a long time because `objcopy` in llvm bin tools was renamed to `llvm-objcopy`. This test was converted into a rmake.rs test, rather straight forward.
2. `tests/run-make/cross-lang-lto-riscv-abi/rmake.rs`: this was broken for a long time and never worked. The old version inspected human-readable output of `llvm-readobj --file-header` looking for substring `EF_RISCV_FLOAT_ABI_DOUBLE`, but the human-readable output will only contain something like `Flags: 0x5, RVC, double-float ABI`, hence it will never match. This test was fixed by instead using the `object` crate to actually decode the ELF headers looking for the specific `e_flags` based on reading the RISCV ELF psABI docs.
This PR is best reviewed commit-by-commit, two commits setup the support library for functionality and two commits are for each of the failing `run-make` tests.
I had to bump the `x86_64-gnu-debug` job to be ran with a runner with larger disk space.
Part of #132034.
try-job: x86_64-gnu-debug
ci: aarch64-gnu-debug job
- Adds a new CI job which checks that the compiler builds with `--enable-debug` and tests that `needs-force-clang-based-tests` pass (where cross-language LTO is tested).
- Add a test confirming that `-Zbranch-protection=pac-ret` and cross-language LTO work together.
r? `@Mark-Simulacrum`
try-job: aarch64-gnu-debug
Adds a new CI job which checks that the compiler builds with
`--enable-debug` and tests that `needs-force-clang-based-tests` pass
(where cross-language LTO is tested).
Add a test case for #131164
The upstream has already been fixed, but it won't be backported to LLVM 19.
r? jieyouxu or compiler
try-job: x86_64-gnu-stable
Fixup Windows verbatim paths when used with the `include!` macro
On Windows, the following code can fail if the `OUT_DIR` environment variable is a [verbatim path](https://doc.rust-lang.org/std/path/enum.Prefix.html) (i.e. begins with `\\?\`):
```rust
include!(concat!(env!("OUT_DIR"), "/src/repro.rs"));
```
This is because verbatim paths treat `/` literally, as if it were just another character in the file name.
The good news is that the standard library already has code to fix this. We can simply use `components` to normalize the path so it works as intended.
Regression test for AVR `rjmp` offset
This adds a regression test for #129301 by minimizing the code in the linked issue and putting it into a `#![no_core]`-compatible format, so that it can easily be used within an `rmake`-test. This needs to be a `rmake` test (opposed to a `tests/assembly` one), since the linked issue describes, that the problem only occurs if the code is directly compiled. Note, that `lld` is used instead of `avr-gcc`; see the [comments](https://github.com/rust-lang/rust/pull/131755#issuecomment-2416469675) [below](https://github.com/rust-lang/rust/pull/131755#issuecomment-2417160045).
Closes#129301.
To show, that the test actually catches the wrong behavior, this can be tested with a faulty rustc:
```bash
$ rustup install nightly-2024-08-19
$ rustc +nightly-2024-08-19 -C opt-level=s -C panic=abort --target avr-unknown-gnu-atmega328 -Clinker=build/x86_64-unknown-linux-gnu/ci-llvm/bin/lld -Clink-arg='--entry=main' -o compiled tests/run-make/avr-rjmp-offset/avr-rjmp-offsets.rs
$ llvm-objdump -d compiled | grep '<main>' -A 6
000110b4 <main>:
110b4: 81 e0 ldi r24, 0x1
110b6: 92 e0 ldi r25, 0x2
110b8: 85 b9 out 0x5, r24
110ba: 95 b9 out 0x5, r25
110bc: fe cf rjmp .-4
```
One can see, that the wrong label offset (`4` instead of `6`) is produced, which would trigger an assertion in the test case.
This would be a good candidate for using the `minicore` proposed in #130693. Since this is not yet merged, there is a FIXME.
r? Patryk27
I think, you are the yet-to-be official target maintainer, hence I'll assign to you.
`@rustbot` label +O-AVR
This fixes the [build error] caused by the `avr-gcc` (used as linker)
not being available in the Rust CI. This is a viable solution, which
shows the wrong/right behavior and, since no functions from `libgcc` are
called, does not produce errors. This was discussed [here]. Another
small problem is, that `lld` doesn't link the correct startup-code by
default. This is not a problem for this test (since it does not actually
use anything the startup code is needed for (no variables, no stack, no
interrupts)), but this causes the `main`-function to be removed by the
default flag `--gc-sections`. Therefore the `rmake`-driver also adds the
linker flag `--entry=main` to mark the `main`-function as the entry
point and thus preventing it from getting removed. The code would work
on a real AVR device.
[build error]: https://github.com/rust-lang/rust/pull/131755#issuecomment-2415127952
[here]: https://github.com/rust-lang/rust/pull/131755#issuecomment-2416469675
Make destructors on `extern "C"` frames to be executed
This would make the example in #123231 print "Noisy Drop". I didn't mark this as fixing the issue because the behaviour is yet to be spec'ed.
Tracking:
- https://github.com/rust-lang/rust/issues/74990
The new `rmake`-content asserts the exact assembly sequence for the loop
preventing false-negatives if some instructions would change and thus
the label offset might need to change.
Since the `tests/assembly` use `emit=asm`, the issue is not observable
as reported in the linked issue. Therefore the existing test case is
converted and a simple `rmake`-test is added. The test only checks, if
the correct `rjmp`-offset is used.
Fix up-to-date checking for run-make tests
This special case in `output_base_dir` had the unfortunate side-effect of causing all run-make tests to share the same `stamp` file. So as soon as any one of them succeeded, all of the failed tests would be incorrectly considered up-to-date and would no longer run in subsequent test invocations.
Fixes#129971.
r? jieyouxu
This special case in `output_base_dir` had the unfortunate side-effect of
causing all run-make tests to share the same `stamp` file. So as soon as any
one of them succeeded, all of the failed tests would be considered up-to-date
and would no longer run in subsequent test invocations.
Create `_imp__` symbols also when doing ThinLTO
When generating a rlib crate on Windows we create `dllimport` / `_imp__` symbols for each global. This effectively makes the rlib contain an import library for itself and allows them to both be dynamically and statically linked. However when doing ThinLTO we do not generate these and thus we end up with missing symbols. Microsoft's `link` can fix these up (and emits warnings), but `lld` seems to currently be unable to.
This PR also does this generation for ThinLTO avoiding those issues with `lld` and also avoids the warnings on `link`.
This is an workaround for https://github.com/rust-lang/rust/issues/81408.
cc `@lqd`
Ignore broken-pipe-no-ice on apple (specifically macOS) for now
This test fails for me locally (initially reported by Zalathar) because apparently on macOS it doesn't say "internal compiler error" but it does report the std I/O panic, and it doesn't exit with a code of 101 but instead terminates with a wait signal of SIGPIPE.
Ignore this test on apple for now, until we try to actually address the underlying issue.
See https://github.com/rust-lang/rust/pull/131155 and https://github.com/rust-lang/rust/issues/131436 for more context.
Prevent building cargo from invalidating build cache of other tools due to conditionally applied `-Zon-broken-pipe=kill` via tracked `RUSTFLAGS`
This PR fixes#130980 where building cargo invalidated the tool build caches of other tools (such as rustdoc) because `-Zon-broken-pipe=kill` was conditionally passed via `RUSTFLAGS` for other tools *except* for cargo. The differing `RUSTFLAGS` triggered tool build cache invalidation as `RUSTFLAGS` is a tracked env var -- any changes in `RUSTFLAGS` requires a rebuild.
`-Zon-broken-pipe=kill` is load-bearing for rustc and rustdoc to not ICE on broken pipes due to usages of raw std `println!` that panics without the flag being set, which manifests in ICEs.
I can't say I like the changes here, but it is what it is...
See detailed discussions and history of `-Zon-broken-pipe=kill` usage in https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Internal.20lint.20for.20raw.20.60print!.60.20and.20.60println!.60.3F/near/474593815.
## Approach
This PR fixes the tool build cache invalidation by informing the `rustc` binary shim when to apply `-Zon-broken-pipe=kill` (i.e. when the rustc binary shim is not used to build cargo). This information is not communicated by `RUSTFLAGS`, which is an env var tracked by cargo, and instead uses an untracked env var `UNTRACKED_BROKEN_PIPE_FLAG` so we won't trigger tool build cache invalidation. We preserve bootstrap's behavior of not setting that flag for cargo by conditionally omitting setting `UNTRACKED_BROKEN_PIPE_FLAG` when building cargo.
Notably, the `-Zon-broken-pipe=kill` instance in 1e5719bdc4/src/bootstrap/src/core/build_steps/compile.rs (L1058) is not modified because that is used to build rustc only and not cargo itself.
Thanks to `@cuviper` for the idea!
## Testing
### Integration testing
This PR introduces a run-make test for rustc and rustdoc that checks that when they do not ICE/panic when they encounter a broken pipe of the stdout stream.
I checked this test will catch the broken pipe ICE regression for rustc on Linux (at least) by commenting out 1e5719bdc4/src/bootstrap/src/core/build_steps/compile.rs (L1058), and the test failed because rustc ICE'd.
### Manual testing
I have manually tried:
1. `./x clean && `./x test build --stage 1` -> `rustc +stage1 --print=sysroot | false`: no ICE.
2. `./x clean` -> `./x test run-make` twice: no stage 1 cargo rebuilds.
3. `./x clean` -> `./x build rustdoc` -> `rustdoc +stage1 --version | false`: no panics.
4. `./x test src/tools/cargo`: tests pass, notably `build::close_output` and `cargo_command::closed_output_ok` do not fail which would fail if cargo was built with `-Zon-broken-pipe=kill`.
## Related discussions
Thanks to everyone who helped!
- https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/Applying.20.60-Zon-broken-pipe.3Dkill.60.20flags.20in.20bootstrap.3F
- https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/Modifying.20run-make.20tests.20unnecessarily.20rebuild.20stage.201.20.2E.2E.2E
- https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Internal.20lint.20for.20raw.20.60print!.60.20and.20.60println!.60.3F
Fixes https://github.com/rust-lang/rust/issues/130980
Closes https://github.com/rust-lang/rust/issues/131059
---
try-job: aarch64-apple
try-job: x86_64-msvc
try-job: x86_64-mingw
Add tests for some old fixed issues
Closes#30867Closes#30472Closes#28994Closes#26719 (and migrates the relevant test to the new run-make)
Closes#23600
cc `@jieyouxu` for the run-make-support changes
try-job: x86_64-msvc
- fix for divergence
- fix error message
- fix another cranelift test
- fix some cranelift things
- don't set the NORETURN option for naked asm
- fix use of naked_asm! in doc comment
- fix use of naked_asm! in run-make test
- use `span_bug` in unreachable branch
Add unstable support for outputting file checksums for use in cargo
Adds an unstable option that appends file checksums and expected lengths to the end of the dep-info file such that `cargo` can read and use these values as an alternative to file mtimes.
This PR powers the changes made in this cargo PR https://github.com/rust-lang/cargo/pull/14137
Here's the tracking issue for the cargo feature https://github.com/rust-lang/cargo/issues/14136.
Apple: Do not specify an SDK version in `rlib` object files
This was added in https://github.com/rust-lang/rust/pull/114114, but is unnecessary, since it ends up being overwritten when linking anyhow, and it feels wrong to embed some arbitrary SDK version in here. The object files produced by LLVM also do not set this, and the tooling shows `n/a` when it's `0`, so it seems to genuinely be optional in object files.
I've also added a test for the different places the SDK version shows up, and documented a bit more in the code how SDK versions work.
See https://github.com/rust-lang/rust/issues/129432 for the bigger picture.
Tested with (excludes the same few targets as in https://github.com/rust-lang/rust/pull/130435):
```console
./x test tests/run-make/apple-sdk-version --target aarch64-apple-darwin,aarch64-apple-ios,aarch64-apple-ios-macabi,aarch64-apple-ios-sim,aarch64-apple-tvos,aarch64-apple-tvos-sim,aarch64-apple-visionos,aarch64-apple-visionos-sim,aarch64-apple-watchos,aarch64-apple-watchos-sim,arm64_32-apple-watchos,armv7k-apple-watchos,armv7s-apple-ios,x86_64-apple-darwin,x86_64-apple-ios,x86_64-apple-ios-macabi,x86_64-apple-tvos,x86_64-apple-watchos-sim,x86_64h-apple-darwin
IPHONEOS_DEPLOYMENT_TARGET=10.0 ./x test tests/run-make/apple-sdk-version --target=i386-apple-ios
```
CC `@BlackHoleFox,` you [originally commented on these values](https://github.com/rust-lang/rust/pull/114114#discussion_r1300599445).
`@rustbot` label O-apple
