Auto merge of #10596 - blyxyas:book-write_tests, r=flip1995
Clippy Book Chapter Updates Reborn: Writing tests This PR adds a new chapter to the book: "Writing tests". The changes have been mainly done from reviews from #9426 and some minor re-writes. ## Notes - We still need to check that the `git status`es are correct, as `cargo dev new_lint` changed a lot since 2022. - Requires #10598: Link to "Emitting Lints" where I flagged with `FIXME:`. - To talk about the whole project, please use the tracking issue for the project #10597 (It also contains a timeline, discussions and more information) changelog: Add a new "Writing tests" chapter to the book r? `@flip1995`
This commit is contained in:
bors
commit
aa371eb154
@ -14,6 +14,7 @@
|
||||
- [Basics](development/basics.md)
|
||||
- [Adding Lints](development/adding_lints.md)
|
||||
- [Defining Lints](development/defining_lints.md)
|
||||
- [Writing tests](development/writing_tests.md)
|
||||
- [Lint Passes](development/lint_passes.md)
|
||||
- [Type Checking](development/type_checking.md)
|
||||
- [Method Checking](development/method_checking.md)
|
||||
|
||||
223
book/src/development/writing_tests.md
Normal file
223
book/src/development/writing_tests.md
Normal file
@ -0,0 +1,223 @@
|
||||
# Testing
|
||||
|
||||
Developing lints for Clippy is a Test-Driven Development (TDD) process because
|
||||
our first task before implementing any logic for a new lint is to write some test cases.
|
||||
|
||||
## Develop Lints with Tests
|
||||
|
||||
When we develop Clippy, we enter a complex and chaotic realm full of
|
||||
programmatic issues, stylistic errors, illogical code and non-adherence to convention.
|
||||
Tests are the first layer of order we can leverage to define when and where
|
||||
we want a new lint to trigger or not.
|
||||
|
||||
Moreover, writing tests first help Clippy developers to find a balance for
|
||||
the first iteration of and further enhancements for a lint.
|
||||
With test cases on our side, we will not have to worry about over-engineering
|
||||
a lint on its first version nor missing out some obvious edge cases of the lint.
|
||||
This approach empowers us to iteratively enhance each lint.
|
||||
|
||||
## Clippy UI Tests
|
||||
|
||||
We use **UI tests** for testing in Clippy. These UI tests check that the output
|
||||
of Clippy is exactly as we expect it to be. Each test is just a plain Rust file
|
||||
that contains the code we want to check.
|
||||
|
||||
The output of Clippy is compared against a `.stderr` file. Note that you don't
|
||||
have to create this file yourself. We'll get to generating the `.stderr` files
|
||||
with the command [`cargo bless`](#cargo-bless) (seen later on).
|
||||
|
||||
### Write Test Cases
|
||||
|
||||
Let us now think about some tests for our imaginary `foo_functions` lint. We
|
||||
start by opening the test file `tests/ui/foo_functions.rs` that was created by
|
||||
`cargo dev new_lint`.
|
||||
|
||||
Update the file with some examples to get started:
|
||||
|
||||
```rust
|
||||
#![warn(clippy::foo_functions)] // < Add this, so the lint is guaranteed to be enabled in this file
|
||||
|
||||
// Impl methods
|
||||
struct A;
|
||||
impl A {
|
||||
pub fn fo(&self) {}
|
||||
pub fn foo(&self) {} //~ ERROR: function called "foo"
|
||||
pub fn food(&self) {}
|
||||
}
|
||||
|
||||
// Default trait methods
|
||||
trait B {
|
||||
fn fo(&self) {}
|
||||
fn foo(&self) {} //~ ERROR: function called "foo"
|
||||
fn food(&self) {}
|
||||
}
|
||||
|
||||
// Plain functions
|
||||
fn fo() {}
|
||||
fn foo() {} //~ ERROR: function called "foo"
|
||||
fn food() {}
|
||||
|
||||
fn main() {
|
||||
// We also don't want to lint method calls
|
||||
foo();
|
||||
let a = A;
|
||||
a.foo();
|
||||
}
|
||||
```
|
||||
|
||||
Without actual lint logic to emit the lint when we see a `foo` function name,
|
||||
this test will just pass, because no lint will be emitted. However, we can now
|
||||
run the test with the following command:
|
||||
|
||||
```sh
|
||||
$ TESTNAME=foo_functions cargo uitest
|
||||
```
|
||||
|
||||
Clippy will compile and it will conclude with an `ok` for the tests:
|
||||
|
||||
```
|
||||
...Clippy warnings and test outputs...
|
||||
test compile_test ... ok
|
||||
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.48s
|
||||
```
|
||||
|
||||
This is normal. After all, we wrote a bunch of Rust code but we haven't really
|
||||
implemented any logic for Clippy to detect `foo` functions and emit a lint.
|
||||
|
||||
As we gradually implement our lint logic, we will keep running this UI test command.
|
||||
Clippy will begin outputting information that allows us to check if the output is
|
||||
turning into what we want it to be.
|
||||
|
||||
### Example output
|
||||
|
||||
As our `foo_functions` lint is tested, the output would look something like this:
|
||||
|
||||
```
|
||||
failures:
|
||||
|
||||
---- compile_test stdout ----
|
||||
normalized stderr:
|
||||
error: function called "foo"
|
||||
--> $DIR/foo_functions.rs:6:12
|
||||
|
|
||||
LL | pub fn foo(&self) {}
|
||||
| ^^^
|
||||
|
|
||||
= note: `-D clippy::foo-functions` implied by `-D warnings`
|
||||
|
||||
error: function called "foo"
|
||||
--> $DIR/foo_functions.rs:13:8
|
||||
|
|
||||
LL | fn foo(&self) {}
|
||||
| ^^^
|
||||
|
||||
error: function called "foo"
|
||||
--> $DIR/foo_functions.rs:19:4
|
||||
|
|
||||
LL | fn foo() {}
|
||||
| ^^^
|
||||
|
||||
error: aborting due to 3 previous errors
|
||||
```
|
||||
|
||||
Note the *failures* label at the top of the fragment, we'll get rid of it
|
||||
(saving this output) in the next section.
|
||||
|
||||
> _Note:_ You can run multiple test files by specifying a comma separated list:
|
||||
> `TESTNAME=foo_functions,bar_methods,baz_structs`.
|
||||
|
||||
### `cargo bless`
|
||||
|
||||
Once we are satisfied with the output, we need to run this command to
|
||||
generate or update the `.stderr` file for our lint:
|
||||
|
||||
```sh
|
||||
$ TESTNAME=foo_functions cargo uibless
|
||||
```
|
||||
|
||||
This writes the emitted lint suggestions and fixes to the `.stderr` file, with
|
||||
the reason for the lint, suggested fixes, and line numbers, etc.
|
||||
|
||||
Running `TESTNAME=foo_functions cargo uitest` should pass then. When we commit
|
||||
our lint, we need to commit the generated `.stderr` files, too.
|
||||
|
||||
In general, you should only commit files changed by `cargo bless` for the
|
||||
specific lint you are creating/editing.
|
||||
|
||||
> _Note:_ If the generated `.stderr`, and `.fixed` files are empty,
|
||||
> they should be removed.
|
||||
|
||||
## `toml` Tests
|
||||
|
||||
Some lints can be configured through a `clippy.toml` file. Those configuration
|
||||
values are tested in `tests/ui-toml`.
|
||||
|
||||
To add a new test there, create a new directory and add the files:
|
||||
|
||||
- `clippy.toml`: Put here the configuration value you want to test.
|
||||
- `lint_name.rs`: A test file where you put the testing code, that should see a
|
||||
different lint behavior according to the configuration set in the
|
||||
`clippy.toml` file.
|
||||
|
||||
The potential `.stderr` and `.fixed` files can again be generated with `cargo
|
||||
bless`.
|
||||
|
||||
## Cargo Lints
|
||||
|
||||
The process of testing is different for Cargo lints in that now we are
|
||||
interested in the `Cargo.toml` manifest file. In this case, we also need a
|
||||
minimal crate associated with that manifest. Those tests are generated in
|
||||
`tests/ui-cargo`.
|
||||
|
||||
Imagine we have a new example lint that is named `foo_categories`, we can run:
|
||||
|
||||
```sh
|
||||
$ cargo dev new_lint --name=foo_categories --pass=late --category=cargo
|
||||
```
|
||||
|
||||
After running `cargo dev new_lint` we will find by default two new crates,
|
||||
each with its manifest file:
|
||||
|
||||
* `tests/ui-cargo/foo_categories/fail/Cargo.toml`: this file should cause the
|
||||
new lint to raise an error.
|
||||
* `tests/ui-cargo/foo_categories/pass/Cargo.toml`: this file should not trigger
|
||||
the lint.
|
||||
|
||||
If you need more cases, you can copy one of those crates (under
|
||||
`foo_categories`) and rename it.
|
||||
|
||||
The process of generating the `.stderr` file is the same as for other lints
|
||||
and prepending the `TESTNAME` variable to `cargo uitest` works for Cargo lints too.
|
||||
|
||||
## Rustfix Tests
|
||||
|
||||
If the lint you are working on is making use of structured suggestions,
|
||||
[`rustfix`] will apply the suggestions from the lint to the test file code and
|
||||
compare that to the contents of a `.fixed` file.
|
||||
|
||||
Structured suggestions tell a user how to fix or re-write certain code that has
|
||||
been linted with [`span_lint_and_sugg`].
|
||||
|
||||
Should `span_lint_and_sugg` be used to generate a suggestion, but not all
|
||||
suggestions lead to valid code, you can use the `//@no-rustfix` comment on top
|
||||
of the test file, to not run `rustfix` on that file.
|
||||
|
||||
We'll talk about suggestions more in depth in a later chapter.
|
||||
<!-- FIXME: (blyxyas) Link to "Emitting lints" when that gets merged -->
|
||||
|
||||
Use `cargo bless` to automatically generate the `.fixed` file after running
|
||||
the tests.
|
||||
|
||||
[`rustfix`]: https://github.com/rust-lang/rustfix
|
||||
[`span_lint_and_sugg`]: https://doc.rust-lang.org/beta/nightly-rustc/clippy_utils/diagnostics/fn.span_lint_and_sugg.html
|
||||
|
||||
## Testing Manually
|
||||
|
||||
Manually testing against an example file can be useful if you have added some
|
||||
`println!`s and the test suite output becomes unreadable.
|
||||
|
||||
To try Clippy with your local modifications, run from the working copy root.
|
||||
|
||||
```sh
|
||||
$ cargo dev lint input.rs
|
||||
```
|
||||
Loading…
Reference in New Issue
Block a user