add -Zmiri-strict-provenance
This implements [strict provenance](https://github.com/rust-lang/rust/issues/95228) in Miri. The only change is that casting an integer to a pointer does not even attempt to produce a good provenance for the given address; instead, it always uses the invalid provenance. This stricter than even `-Zmiri-tag-raw-pointers` in that it also rejects the following example (which does not even involve Stacked Borrows):
```rust
fn main() {
let x = 22;
let ptr = &x as *const _ as *const u8;
let roundtrip = ptr as usize as *const u8;
let _ = unsafe { roundtrip.offset(1) };
}
```
The new flag also implies `-Zmiri-tag-raw-pointers` since the only reason one would *not* want to tag raw pointers is to support ptr-int-ptr roundtrips.
Note that the flag does *not* check against ptr-to-int *transmutes*; that still requires `-Zmiri-check-number-validity`. You can also check for strict provenance *without* Stacked Borrows by adding `-Zmiri-disable-stacked-borrows`.
The new "Miri hard mode" flags for maximal checking are `-Zmiri-strict-provenance -Zmiri-check-number-validity`. (Add `-Zmiri-symbolic-alignment-check` if you feel extra spicy today.)
Make backtraces work with #[global_allocator]
Currently, backtraces break when the global allocator is overridden because the allocator will attempt to deallocate memory allocated directly by Miri.
~~This PR fixes that by using a new memory kind and providing a function to deallocate it. We can't call the custom allocator to allocate because it's not possible to call a function in the middle of a shim.~~
This PR fixes that by adding a new version of the backtrace API accessible by setting `flags` to 1. Existing code still functions.
backtrace-rs PR: rust-lang/backtrace-rs#462
Fixes https://github.com/rust-lang/miri/issues/1996
Add a lot more information to SB fatal errors
In fatal errors, this clarifies the difference between a tag not being present in the borrow stack at all, and the tag being present but granting SRO. It also introduces a little notation for memory ranges so we can mention to the user that the span may point to code that operates on multiple memory locations, but we are reporting an error at a particular offset.
This also gets rid of the unqualified phrase "the borrow stack" in errors, and clarifies that it is the borrow stack _for some location_.
The crate `pdqselect` v0.1.1:
Before:
```
2103 | unsafe { copy_nonoverlapping(src, dst, count) }
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ no item granting read access to tag <2357> at alloc1029 found in borrow stack.
```
After:
```
2103 | unsafe { copy_nonoverlapping(src, dst, count) }
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| |
| attempting a read access using <2357> at alloc1029[0x0], but that tag does not exist in the borrow stack for this location
| this error occurs as part of an access at alloc1029[0x0..0x4]
```
And the crate `half` v1.8.2
Before:
```
131 | unsafe { &mut *ptr::slice_from_raw_parts_mut(data, len) }
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ trying to reborrow for Unique at alloc1051, but parent tag <2091> does not have an appropriate item in the borrow stack
```
After:
```
131 | unsafe { &mut *ptr::slice_from_raw_parts_mut(data, len) }
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| |
| trying to reborrow <2091> for Unique permission at alloc1051[0x0], but that tag only grants SharedReadOnly permission for this location
| this error occurs as part of a reborrow at alloc1051[0x0..0x6]
```
This tries to clarify exactly why an access is not valid by printing
what memory range the access was over, which in combination with
tag-tracking may help a user figure out the source of the problem.
Allow varargs for libc::open when it is allowed by the second argument
This PR allows `libc::open` to be called using two or three arguments as defined in https://man7.org/linux/man-pages/man2/open.2.html
The presence of the third argument depends on the value of the second argument. If the second argument dictates that the third argument is *required* miri will emit an error if the argument is missing. If the second argument does *not* require a third argument, then the argument is ignored and passed as 0 internally (it would be ignored by libc anyway)
add flag to forward specific env vars (while isolation remains enabled)
The flag is called `-Zmiri-env-forward=<var>`, but I am open to bikeshedding. ;)
exclude mutable references to !Unpin types from uniqueness guarantees
This basically works around https://github.com/rust-lang/unsafe-code-guidelines/issues/148 by not requiring uniqueness any more for mutable references to self-referential generators. That corresponds to [the same work-around that was applied in rustc itself](b815532674/compiler/rustc_middle/src/ty/layout.rs (L2482)).
I am not entirely sure if this is a good idea since it might hide too many errors in case types are "accidentally" `!Unpin`. OTOH, our test suite still passes, and to my knowledge the vast majority of types is `Unpin`. (`place.layout.ty` is monomorphic, we should always exactly know which type this is.)
intptrcast: Never allocate two objects directly adjecent
When two objects directly follow each other in memory, what is the
provenance of an integer cast to a pointer that points directly between
them? For a zero-size region, it could point into the end of the first
object, or the start of the second.
We can avoid answering this difficult question by simply never
allocating two objects directly beside each other. This fixes some of
the false positives from #1866.
When two objects directly follow each other in memory, what is the
provenance of an integer cast to a pointer that points directly between
them? For a zero-size region, it could point into the end of the first
object, or the start of the second.
We can avoid answering this difficult question by simply never
allocating two objects directly beside each other. This fixes some of
the false positives from #1866.
also ignore 'thread leaks' with -Zmiri-ignore-leaks
This is a step towards https://github.com/rust-lang/miri/issues/1371. The remaining hard part would be supporting checking for memory leaks when there are threads still running. For now we elegantly avoid this problem by using the same flag to control both of these checks. :)
Change the code to either `EACCES` (if the op is performed on the
path), or `EBADF` (if the op is performed the fd)
Updated ops: `stat`, `opendir`, `ftruncate64`, and `readlink`
Add a new test for fs ops in isolation.
Report an error if a `#[no_mangle]`/`#[export_name = ...]` function has the same symbol name as a built-in shim
Implements https://github.com/rust-lang/miri/pull/1776#issuecomment-821322605.
The error looks like this:
```
error: found `malloc` symbol definition that clashes with a built-in shim
--> tests/compile-fail/function_calls/exported_symbol_shim_clashing.rs:12:9
|
12 | malloc(0);
| ^^^^^^^^^ found `malloc` symbol definition that clashes with a built-in shim
|
help: the `malloc` symbol is defined here
--> tests/compile-fail/function_calls/exported_symbol_shim_clashing.rs:2:1
|
2 | / extern "C" fn malloc(_: usize) -> *mut std::ffi::c_void {
3 | | //~^ HELP the `malloc` symbol is defined here
4 | | unreachable!()
5 | | }
| |_^
= note: inside `main` at tests/compile-fail/function_calls/exported_symbol_shim_clashing.rs:12:9
```
This does not implement "better error messages than we do currently for arg/ABI mismatches" in https://github.com/rust-lang/miri/pull/1776#issuecomment-821343175 -- I failed to remove all `check_arg_count()` and `check_abi()` (they are still used in `src/shims/intrinsics.rs` and `call_dlsym()`) and they don't receive the name of the shim.
Add support for panicking in the emulated application when unsupported functionality is encountered
This PR fixes#1807 and allows an optional flag to be specified to panic when an unsupported syscall is encountered. In essence, instead of bubbling up an error in the context of the Miri application Miri will panic within the context of the *emulated* application. This feature is desired to allow CI pipelines to determine if a Miri failure is unsupported functionality or actual UB. Please read [this comment](https://github.com/rust-lang/miri/issues/1807#issuecomment-845425076) for the rationale behind this change.
Note: this change does not cover all cases where unsupported functionality errors may be raised. If you search the repo for `throw_unsup_format!` there are many cases that I think are less likely to occur and may still be problematic for some folks.
TODO:
- [x] README documentation on this new flag
- [x] Add tests
In user interface, added a new flag `-Zmiri-isolation-error` which
takes one of the four values -- hide, warn, warn-nobacktrace, and
abort. This option can be used to configure Miri to either abort or
return an error code upon executing isolated op. If not aborted, Miri
prints a warning, whose verbosity can be configured using this flag.
In implementation, added a new enum `IsolatedOp` to capture all the
settings related to ops requiring communication with the
host. Old `communicate` flag in both miri configs and machine
stats is replaced with a new helper function `communicate()` which
checks `isolated_op` internally.
Added a new helper function `reject_in_isolation` which can be called
by shims to reject ops according to the reject_with settings. Use miri
specific diagnostics function `report_msg` to print backtrace in the
warning. Update it to take an enum value instead of a bool, indicating
the level of diagnostics.
Updated shims related to current dir to use the new APIs. Added a new
test for current dir ops in isolation without halting machine.
Add atomic min and max
Closes#1718
Previous attempt: #1653
TODO:
- [x] Merge `atomic_op` and `atomic_min_max` functions
- [x] Fix CI
**Note:** this PR also removes arbitrary trailing whitespace and generally formats the affected files
Add random failures to compare_exchange_weak
In practice this is pretty useful for detecting bugs.
This fails more frequently than realistic (~~50%~~ (now 80%, controlled by a flag) of the time). I couldn't find any existing code that tries to model this (tsan, cdschecker, etc all seem to have TODOs there). Relacy models it with a 25% or 50% failure chance depending on some settings.
CC `@JCTyblaidd` who wrote the code this modifies initially, and seems interested in this subject.
Weaken release sequences to match the C++20 memory model
See [Weaken Release Sequences](http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0982r1.html), since the exception for relaxed writes on the same thread as a release write not blocking release sequences was removed in the C++20 memory model compared to the C11 memory model the paper was based on. The implementation can be updated and simplified to match this. [Rust is currently specified to use the C++20 memory model](https://doc.rust-lang.org/std/sync/atomic/index.html).
Add simple data-race detector
Partially fixes data-race detection, see #1372, based on Dynamic Race Detection for C++11
- This does not explore weak memory behaviour, only exploring one sequentially consistent ordering.
- Data-race detection is only enabled after the first thread is created, so should have minimal overhead for non-concurrent execution.
- ~~Does not attempt to re-use thread id's so creating and joining threads lots of time in an execution will result in the vector clocks growing in size and slowing down program execution~~ It does now
The `backtrace-rs` crate can use this to implement
`Frame::symbol_address`, which is used to skip frames
above the call to `Backtrace::capture` on the stack.
The function pointer will not be useable for comparison purposes if the
function is generic, as CTFE creates a new function pointer for each
cast of a (monomorphized) generic function. However, this already
affects code running under Miri, and isn't a problem for `backtrace-rs`
(which only casts a non-generic function).
I've added logic to allow `MiriFrame` to have either 4 or 5 fields - if
a 5th field is present, we write the function pointer to it.
This PR adds two new Miri-defined extern functions:
`miri_get_backtrace` and `miri_resolve_frame`, which are documented in
the README. Together, they allow obtaining a backtrace for the currently
executing program.
I've added a test showing how these APIs are used. I've also prepared a
companion PR `backtrace-rs`, which will allow
`backtrace::Backtrace::new()` to work automatically under Miri.
Once these two PRs are merged, we will be able to print backtraces from
the normal Rust panic hook (since libstd is now using backtrace-rs).
A few notes:
* Resolving the backtrace frames is *very* slow - you can actually see
each line being printed out one at a time. Some local testing showed
that this is not (primrary) caused by resolving a `Span` - it seems
to be just Miri being slow.
* For the first time, we now interact directly with a user-defined
struct (instead of just executing the user-provided MIR that
manipulates the struct). To allow for future changes, I've added
a 'version' parameter (currently required to be 0). This should allow
us to change the `MiriFrame` struct should the need ever arise.
* I used the approach suggested by @oli-obk - a returned backtrace
pointer consists of a base function allocation, with the 'offset'
used to encode the `Span.lo`. This allows losslessly reconstructing
the location information in `miri_resolve_frame`.
* There are a few quirks on the `backtrace-rs` side:
* `backtrace-rs` calls `getcwd()` by default to try to simplify
the filename. This results in an isolation error by default,
which could be annoying when printing a backtrace from libstd.
* `backtrace-rs` tries to remove 'internal' frames (everything between
the call to `Backtrace::new()` and the internal API call made by
backtrace-rs) by comparing the returned frame pointer value to
a Rust function pointer. This doesn't work due to the way we
construct the frame pointers passed to the caller. We could
attempt to support this kind of comparison, or just add a
`#[cfg(miri)]` and ignore the frames ourselves.
Support F_DUPFD on stdin/stdout/stderr
Enable `close`-ing stdin/stdout/stderr
For `dup`, check if FD is `File` first
If not, clone the appropriate standard IO stream
Merge POSIX `close` and `dup` tests into same module
Also, add assertion that `write` on a closed FD returns an error.
Add `dup` as FileDescriptor trait fn
Also:
- Fix `close` so it drops `self` instead of reference to it
- Remove FD clamping in insert_fd_with_min_fd, since FDs 0-2 can be
closed
Fix fs_libc tests
Make error message when closing stdin/out/err more specific
Return io::Result from `FileDescriptor::dup`
Change error message when closing stdin/out/err
Refactor `FileDescriptor::dup` impl for `FileHandle`
Remove empty line
Nanosleep
This PR adds a shim for `libc::nanosleep`, (available under -Zmiri-disable-isolation only) which backs `thread::sleep` on Linux and macOS. I started off by extracting the `timespec` parsing from the `pthread_cond_timedwait` shim into a helper method, and adding checks for invalid values. The second commit adds the new shim and a small test. The shim blocks the current thread, and registers a timeout callback to unblock the thread again, using the same method as `pthread_cond_timedwait` does.
Move panic payload state from Machine to Thread
This PR moves the panic payload storage from the `Machine` state to per-thread state. Prior to this change, if one thread panicked while another was still unwinding, Miri would fail with `thread 'rustc' panicked at 'the panic runtime should avoid double-panics', src/shims/panic.rs:51:9`. I ran into this issue while prototyping a round-robin scheduler, but it's also reachable with the current scheduler and contrived programs that use blocking API calls to cause thread switching during unwinding. I wrote a test case along those lines for this change.
Re-organize platform-specific shims
Move platform-specific code to `shims::{posix::{linux, macos}, windows}`. Also make it private in these modules to ensure we are reasonably structured.
Add file sync shims
This PR adds shim implementations for these related file syncing functions.
* `fsync`, for POSIX targets, backed by `File::sync_all()`
* `fdatasync`, for POSIX targets, backed by `File::sync_data()`
* `fcntl` with command `F_FULLFSYNC`, for macOS targets, backed by `File::sync_all()`
* `sync_file_range`, for Linux targets, backed by `File::sync_data()`
Synchronization primitive cleanup
Make some methods infallible, move a bit more work into the platform-independent `sync.rs`, and fix a bug in rwlock unlocking.
