# Compiler Test Documentation In the Rust project, we use a special set of commands embedded in comments to test the Rust compiler. There are two groups of commands: 1. Header commands 2. Error info commands Both types of commands are inside comments, but header commands should be in a comment before any code. ## Summary of Error Info Commands Error commands specify something about certain lines of the program. They tell the test what kind of error and what message you are expecting. * `~`: Associates the following error level and message with the current line * `~|`: Associates the following error level and message with the same line as the previous comment * `~^`: Associates the following error level and message with the previous line. Each caret (`^`) that you add adds a line to this, so `~^^^^^^^` is seven lines up. The error levels that you can have are: 1. `ERROR` 2. `WARNING` 3. `NOTE` 4. `HELP` and `SUGGESTION`* \* **Note**: `SUGGESTION` must follow immediately after `HELP`. ## Summary of Header Commands Header commands specify something about the entire test file as a whole, instead of just a few lines inside the test. * `ignore-X` where `X` is a target detail or stage will ignore the test accordingly (see below) * `ignore-pretty` will not compile the pretty-printed test (this is done to test the pretty-printer, but might not always work) * `ignore-test` always ignores the test * `ignore-lldb` and `ignore-gdb` will skip the debuginfo tests * `min-{gdb,lldb}-version` * `should-fail` indicates that the test should fail; used for "meta testing", where we test the compiletest program itself to check that it will generate errors in appropriate scenarios. This header is ignored for pretty-printer tests. * `gate-test-X` where `X` is a feature marks the test as "gate test" for feature X. Such tests are supposed to ensure that the compiler errors when usage of a gated feature is attempted without the proper `#![feature(X)]` tag. Each unstable lang feature is required to have a gate test. Some examples of `X` in `ignore-X`: * Architecture: `aarch64`, `arm`, `asmjs`, `mips`, `wasm32`, `x86_64`, `x86`, ... * OS: `android`, `emscripten`, `freebsd`, `ios`, `linux`, `macos`, `windows`, ... * Environment (fourth word of the target triple): `gnu`, `msvc`, `musl`. * Pointer width: `32bit`, `64bit`. * Stage: `stage0`, `stage1`, `stage2`. ## Revisions Certain classes of tests support "revisions" (as of the time of this writing, this includes run-pass, compile-fail, run-fail, and incremental, though incremental tests are somewhat different). Revisions allow a single test file to be used for multiple tests. This is done by adding a special header at the top of the file: ``` // revisions: foo bar baz ``` This will result in the test being compiled (and tested) three times, once with `--cfg foo`, once with `--cfg bar`, and once with `--cfg baz`. You can therefore use `#[cfg(foo)]` etc within the test to tweak each of these results. You can also customize headers and expected error messages to a particular revision. To do this, add `[foo]` (or `bar`, `baz`, etc) after the `//` comment, like so: ``` // A flag to pass in only for cfg `foo`: //[foo]compile-flags: -Z verbose #[cfg(foo)] fn test_foo() { let x: usize = 32_u32; //[foo]~ ERROR mismatched types } ``` Note that not all headers have meaning when customized to a revision. For example, the `ignore-test` header (and all "ignore" headers) currently only apply to the test as a whole, not to particular revisions. The only headers that are intended to really work when customized to a revision are error patterns and compiler flags. ## Guide to the UI Tests The UI tests are intended to capture the compiler's complete output, so that we can test all aspects of the presentation. They work by compiling a file (e.g., `ui/hello_world/main.rs`), capturing the output, and then applying some normalization (see below). This normalized result is then compared against reference files named `ui/hello_world/main.stderr` and `ui/hello_world/main.stdout`. If either of those files doesn't exist, the output must be empty. If the test run fails, we will print out the current output, but it is also saved in `build//test/ui/hello_world/main.stdout` (this path is printed as part of the test failure message), so you can run `diff` and so forth. ### Editing and updating the reference files If you have changed the compiler's output intentionally, or you are making a new test, you can use the script `ui/update-references.sh` to update the references. When you run the test framework, it will report various errors: in those errors is a command you can use to run the `ui/update-references.sh` script, which will then copy over the files from the build directory and use them as the new reference. You can also just run `ui/update-all-references.sh`. In both cases, you can run the script with `--help` to get a help message. ### Normalization The normalization applied is aimed at eliminating output difference between platforms, mainly about filenames: - the test directory is replaced with `$DIR` - all backslashes (`\`) are converted to forward slashes (`/`) (for Windows) - all CR LF newlines are converted to LF Sometimes these built-in normalizations are not enough. In such cases, you may provide custom normalization rules using the header commands, e.g. ``` // normalize-stdout-test: "foo" -> "bar" // normalize-stderr-32bit: "fn\(\) \(32 bits\)" -> "fn\(\) \($$PTR bits\)" // normalize-stderr-64bit: "fn\(\) \(64 bits\)" -> "fn\(\) \($$PTR bits\)" ``` This tells the test, on 32-bit platforms, whenever the compiler writes `fn() (32 bits)` to stderr, it should be normalized to read `fn() ($PTR bits)` instead. Similar for 64-bit. The replacement is performed by regexes using default regex flavor provided by `regex` crate. The corresponding reference file will use the normalized output to test both 32-bit and 64-bit platforms: ``` ... | = note: source type: fn() ($PTR bits) = note: target type: u16 (16 bits) ... ``` Please see `ui/transmute/main.rs` and `.stderr` for a concrete usage example. Besides `normalize-stderr-32bit` and `-64bit`, one may use any target information or stage supported by `ignore-X` here as well (e.g. `normalize-stderr-windows` or simply `normalize-stderr-test` for unconditional replacement).