Prevent building cargo from invalidating build cache of other tools due to conditionally applied `-Zon-broken-pipe=kill` via tracked `RUSTFLAGS`
This PR fixes#130980 where building cargo invalidated the tool build caches of other tools (such as rustdoc) because `-Zon-broken-pipe=kill` was conditionally passed via `RUSTFLAGS` for other tools *except* for cargo. The differing `RUSTFLAGS` triggered tool build cache invalidation as `RUSTFLAGS` is a tracked env var -- any changes in `RUSTFLAGS` requires a rebuild.
`-Zon-broken-pipe=kill` is load-bearing for rustc and rustdoc to not ICE on broken pipes due to usages of raw std `println!` that panics without the flag being set, which manifests in ICEs.
I can't say I like the changes here, but it is what it is...
See detailed discussions and history of `-Zon-broken-pipe=kill` usage in https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Internal.20lint.20for.20raw.20.60print!.60.20and.20.60println!.60.3F/near/474593815.
## Approach
This PR fixes the tool build cache invalidation by informing the `rustc` binary shim when to apply `-Zon-broken-pipe=kill` (i.e. when the rustc binary shim is not used to build cargo). This information is not communicated by `RUSTFLAGS`, which is an env var tracked by cargo, and instead uses an untracked env var `UNTRACKED_BROKEN_PIPE_FLAG` so we won't trigger tool build cache invalidation. We preserve bootstrap's behavior of not setting that flag for cargo by conditionally omitting setting `UNTRACKED_BROKEN_PIPE_FLAG` when building cargo.
Notably, the `-Zon-broken-pipe=kill` instance in 1e5719bdc4/src/bootstrap/src/core/build_steps/compile.rs (L1058) is not modified because that is used to build rustc only and not cargo itself.
Thanks to `@cuviper` for the idea!
## Testing
### Integration testing
This PR introduces a run-make test for rustc and rustdoc that checks that when they do not ICE/panic when they encounter a broken pipe of the stdout stream.
I checked this test will catch the broken pipe ICE regression for rustc on Linux (at least) by commenting out 1e5719bdc4/src/bootstrap/src/core/build_steps/compile.rs (L1058), and the test failed because rustc ICE'd.
### Manual testing
I have manually tried:
1. `./x clean && `./x test build --stage 1` -> `rustc +stage1 --print=sysroot | false`: no ICE.
2. `./x clean` -> `./x test run-make` twice: no stage 1 cargo rebuilds.
3. `./x clean` -> `./x build rustdoc` -> `rustdoc +stage1 --version | false`: no panics.
4. `./x test src/tools/cargo`: tests pass, notably `build::close_output` and `cargo_command::closed_output_ok` do not fail which would fail if cargo was built with `-Zon-broken-pipe=kill`.
## Related discussions
Thanks to everyone who helped!
- https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/Applying.20.60-Zon-broken-pipe.3Dkill.60.20flags.20in.20bootstrap.3F
- https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/Modifying.20run-make.20tests.20unnecessarily.20rebuild.20stage.201.20.2E.2E.2E
- https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Internal.20lint.20for.20raw.20.60print!.60.20and.20.60println!.60.3F
Fixes https://github.com/rust-lang/rust/issues/130980
Closes https://github.com/rust-lang/rust/issues/131059
---
try-job: aarch64-apple
try-job: x86_64-msvc
try-job: x86_64-mingw
The run-make test suite contains tests which are the most flexible out of all
the rust-lang/rust test suites. run-make
tests can basically contain arbitrary code, and are supported by the
run_make_support library.
Infrastructure
There are two kinds of run-make tests:
The new rmake.rs version: this allows run-make tests to be written in Rust
(with rmake.rs as the main test file).
The legacy Makefile version: this is what run-make tests were written with
before support for rmake.rs was introduced.
The implementation for collecting and building the rmake.rs recipes (or
Makefiles) are in
src/tools/compiletest/src/runtest.rs,
in run_rmake_v2_test and run_rmake_legacy_test.
Rust-based run-make tests: rmake.rs
The setup for the rmake.rs version is a 3-stage process:
First, we build the run_make_support library in bootstrap as a tool lib.
Then, we compile the rmake.rs "recipe" linking the support library and its
dependencies in, and provide a bunch of env vars. We setup a directory
structure within build/<target>/test/run-make/
<test-name>/
rmake.exe # recipe binary
rmake_out/ # sources from test sources copied over
and copy non-rmake.rs input support files over to rmake_out/. The
support library is made available as an extern prelude.
Finally, we run the recipe binary and set rmake_out/ as the working
directory.
Formatting
Note that files under tests/ are not formatted by ./x fmt,
use rustfmt tests/path/to/file.rs to format a specific file if desired.
