Improve isatty support
Per https://github.com/rust-lang/miri/issues/2292#issuecomment-1171858283, this is an attempt at
> do something more clever with Miri's `isatty` shim
Since Unix -> Unix is very simple, I'm starting with a patch that just does that. Happy to augment/rewrite this based on feedback.
The linked file in libtest specifically only supports stdout. If we're doing this to support terminal applications, I think it would be strange to support one but not all 3 of the standard streams.
The `atty` crate contains a bunch of extra logic that libtest does not contain, in order to support MSYS terminals: db8d55f88e so I think if we're going to do Windows support, we should probably access all that logic somehow. I think it's pretty clear that the implementation is not going to change, so I think if we want to, pasting the contents of the `atty` crate into Miri is on the table, instead of taking a dependency.
reborrow error: clarify that we are reborrowing *from* that tag
`@saethlin` I found the current message not entirely clear, so what do you think about this?
Optimizing Stacked Borrows (part 2): Shrink Item
This moves protectors out of `Item`, storing them both in a global `HashSet` which contains all currently-protected tags as well as a `Vec<SbTag>` on each `Frame` so that when we return from a function we know which tags to remove from the protected set.
This also bit-packs the 64-bit tag and the 2-bit permission together when they are stored in memory. This means we theoretically run out of tags sooner, but I doubt that limit will ever be hit.
Together these optimizations reduce the memory footprint of Miri when executing programs which stress Stacked Borrows by ~66%. For example, running a test with isolation off which only panics currently peaks at ~19 GB, with this PR it peaks at ~6.2 GB.
To-do
- [x] Enforce the 62-bit limit
- [x] Decide if there is a better order to pack the tag and permission in
- [x] Wait for `UnsafeCell` to become infectious, or express offsets + tags in the global protector set
Benchmarks before:
```
Benchmark 1: cargo +miri miri run --manifest-path bench-cargo-miri/backtraces/Cargo.toml
Time (mean ± σ): 8.948 s ± 0.253 s [User: 8.752 s, System: 0.158 s]
Range (min … max): 8.619 s … 9.279 s 5 runs
Benchmark 1: cargo +miri miri run --manifest-path bench-cargo-miri/mse/Cargo.toml
Time (mean ± σ): 2.129 s ± 0.037 s [User: 1.849 s, System: 0.248 s]
Range (min … max): 2.086 s … 2.176 s 5 runs
Benchmark 1: cargo +miri miri run --manifest-path bench-cargo-miri/serde1/Cargo.toml
Time (mean ± σ): 3.334 s ± 0.017 s [User: 3.211 s, System: 0.103 s]
Range (min … max): 3.315 s … 3.352 s 5 runs
Benchmark 1: cargo +miri miri run --manifest-path bench-cargo-miri/serde2/Cargo.toml
Time (mean ± σ): 3.316 s ± 0.038 s [User: 3.207 s, System: 0.095 s]
Range (min … max): 3.282 s … 3.375 s 5 runs
Benchmark 1: cargo +miri miri run --manifest-path bench-cargo-miri/unicode/Cargo.toml
Time (mean ± σ): 6.391 s ± 0.323 s [User: 5.928 s, System: 0.412 s]
Range (min … max): 6.090 s … 6.917 s 5 runs
```
After:
```
Benchmark 1: cargo +miri miri run --manifest-path bench-cargo-miri/backtraces/Cargo.toml
Time (mean ± σ): 6.955 s ± 0.051 s [User: 6.807 s, System: 0.132 s]
Range (min … max): 6.900 s … 7.038 s 5 runs
Benchmark 1: cargo +miri miri run --manifest-path bench-cargo-miri/mse/Cargo.toml
Time (mean ± σ): 1.784 s ± 0.012 s [User: 1.627 s, System: 0.156 s]
Range (min … max): 1.772 s … 1.797 s 5 runs
Benchmark 1: cargo +miri miri run --manifest-path bench-cargo-miri/serde1/Cargo.toml
Time (mean ± σ): 2.505 s ± 0.095 s [User: 2.311 s, System: 0.096 s]
Range (min … max): 2.405 s … 2.603 s 5 runs
Benchmark 1: cargo +miri miri run --manifest-path bench-cargo-miri/serde2/Cargo.toml
Time (mean ± σ): 2.449 s ± 0.031 s [User: 2.306 s, System: 0.100 s]
Range (min … max): 2.395 s … 2.467 s 5 runs
Benchmark 1: cargo +miri miri run --manifest-path bench-cargo-miri/unicode/Cargo.toml
Time (mean ± σ): 3.667 s ± 0.110 s [User: 3.498 s, System: 0.140 s]
Range (min … max): 3.564 s … 3.814 s 5 runs
```
The decrease in system time is probably due to spending less time in the page fault handler.
stacked_borrow now has an item module, and its own FrameExtra. These
serve to protect the implementation of Item (which is a bunch of
bit-packing tricks) from the primary logic of Stacked Borrows, and the
FrameExtra we have separates Stacked Borrows more cleanly from the
interpreter itself.
The new strategy for checking protectors also makes some subtle
performance tradeoffs, so they are now documented in Stack::item_popped
because that function primarily benefits from them, and it also touches
every aspect of them.
Also separating the actual CallId that is protecting a Tag from the Tag
makes it inconvienent to reproduce exactly the same protector errors, so
this also takes the opportunity to use some slightly cleaner English in
those errors. We need to make some change, might as well make it good.
Previously, Item was a struct of a NonZeroU64, an Option which was
usually unset or irrelevant, and a 4-variant enum. So collectively, the
size of an Item was 24 bytes, but only 8 bytes were used for the most
part.
So this takes advantage of the fact that it is probably impossible to
exhaust the total space of SbTags, and steals 3 bits from it to pack the
whole struct into a single u64. This bit-packing means that we reduce
peak memory usage when Miri goes memory-bound by ~3x. We also get CPU
performance improvements of varying size, because not only are we simply
accessing less memory, we can now compare a Vec<Item> using a memcmp
because it does not have any padding.
fix comparing wide raw pointers
Fixes https://github.com/rust-lang/rust/issues/96169
However I am not sure if these are the correct semantics. I'll wait for confirmation in that issue.
Extend `environ` linux extern implementation to freebsd
This fixes the `env` test on freebsd, and enables the CI test
Signed-off-by: InfRandomness <infrandomness@gmail.com>
Enable permissive provenance by default
This completes the plan laid out in https://github.com/rust-lang/miri/issues/2133:
- We use permissive provenance with wildcard pointers by default.
- We print a warning on int2ptr casts. `-Zmiri-permissive-provenance` suppresses the warning; `-Zmiri-strict-provenance` turns it into a hard error.
- Raw pointer tagging is now always enabled, so we remove the `-Zmiri-tag-raw-pointers` flag and the code for untagged pointers. (Passing the flag still works, for compatibility -- but we just ignore it, with a warning.)
We also fix an intptrcast issue:
- Only live allocations are considered when computing the AllocId from an address.
So, finally, Miri has a good story for ptr2int2ptr roundtrips *and* no weird false negatives when doing raw pointer stuff with Stacked Borrows. :-) 🎉 Thanks a lot to everyone who helped with this, in particular `@carbotaniuman` who convinced me this is even possible.
Fixes https://github.com/rust-lang/miri/issues/2133
Fixes https://github.com/rust-lang/miri/issues/1866
Fixes https://github.com/rust-lang/miri/issues/1993
test that futexes induce appropriate synchronization
This fails when I remove the `validate_lock_acquire` or `validate_lock_release` from `futex_wake`. So finally we got those code paths actually covered in tests. :)
Format tests with rustfmt (288-299 of 299)
Extracted from #2097.
I'll make a separate PR to enable checking the `tests` directory's formatting in CI. I'll need to rebase that after both this and #2254 have landed, and if any new non-rustfmt-formatted files appear in the meantime, we can include formatting those in the same PR that enables the CI.
Format tests with rustfmt (225-275 of 300)
Extracted from #2097.
These cases all involve a line comment at the end of a block that rustfmt has chosen to wrap.
```diff
- unsafe { (*ptr).set(20); } //~ ERROR does not exist in the borrow stack
+ unsafe {
+ (*ptr).set(20);
+ } //~ ERROR does not exist in the borrow stack
```
I have moved all of those comments back onto the same line as the content of the block instead, as was indicated being `@RalfJung's` preference in https://github.com/rust-lang/miri/pull/2097#discussion_r862436672.
```diff
+ unsafe {
+ (*ptr).set(20); //~ ERROR does not exist in the borrow stack
+ }
```
Format tests with rustfmt (151-200 of 300)
Extracted from #2097.
This PR is still only doing the easy cases with no comments involved.
In the next PRs after this, I'll start grouping by common comment patterns, e.g. all the cases resembling https://github.com/rust-lang/miri/pull/2097#discussion_r862436672 together in one PR.
Format tests with rustfmt (51-100 of 300)
Extracted from #2097.
Like #2244, this is intended to be "easy" cases which don't involve comments in the vicinity.
Five of the files being skipped here are because rustfmt is buggy (see
the error messages below). The others have clearly preferable manual
formatting.
error[internal]: left behind trailing whitespace
--> tests/fail/validity/transmute_through_ptr.rs:18:18:1
|
18 |
| ^^^^
|
warning: rustfmt has failed to format. See previous 1 errors.
error[internal]: left behind trailing whitespace
--> tests/fail/stacked_borrows/illegal_read2.rs:10:10:1
|
10 |
| ^^^^
|
warning: rustfmt has failed to format. See previous 1 errors.
error[internal]: left behind trailing whitespace
--> tests/fail/stacked_borrows/illegal_read5.rs:15:15:1
|
15 |
| ^^^^
|
warning: rustfmt has failed to format. See previous 1 errors.
error[internal]: left behind trailing whitespace
--> tests/fail/stacked_borrows/illegal_read1.rs:10:10:1
|
10 |
| ^^^^
|
warning: rustfmt has failed to format. See previous 1 errors.
error[internal]: left behind trailing whitespace
--> tests/fail/erroneous_const2.rs:9:9:1
|
9 |
| ^^^^
|
warning: rustfmt has failed to format. See previous 1 errors.
Prevent futex_wait from actually waiting if a concurrent waker was executed before us
Fixes#2223
Two SC fences were placed in `futex_wake` (after the caller has changed `addr`), and in `futex_wait` (before we read `addr`). This guarantees that `futex_wait` sees the value written to `addr` before the last `futex_wake` call, should one exists, and avoid going into sleep with no one else to wake us up.
ada7b72a87/src/concurrency/weak_memory.rs (L324-L326)
Earlier I proposed to use `fetch_add(0)` to read the latest value in MO, though this isn't the proper way to do it and breaks aliasing: syscall caller may pass in a `*const` from a `&` and Miri complains about write to a `SharedReadOnly` location, causing this test to fail.
ada7b72a87/tests/pass/concurrency/linux-futex.rs (L56-L68)
test ui output also in rustc test suite
`@oli-obk` when I just tried this locally (`./x.py test src/tools/miri --stage 0`), it worked fine. What differences had you seen before?
