diff --git a/book/src/SUMMARY.md b/book/src/SUMMARY.md index daaefd06a97..9df393db8a0 100644 --- a/book/src/SUMMARY.md +++ b/book/src/SUMMARY.md @@ -14,6 +14,7 @@ - [Basics](development/basics.md) - [Adding Lints](development/adding_lints.md) - [Defining Lints](development/defining_lints.md) + - [Writing tests](development/writing_tests.md) - [Lint Passes](development/lint_passes.md) - [Type Checking](development/type_checking.md) - [Method Checking](development/method_checking.md) diff --git a/book/src/development/writing_tests.md b/book/src/development/writing_tests.md new file mode 100644 index 00000000000..7cb530a3bb9 --- /dev/null +++ b/book/src/development/writing_tests.md @@ -0,0 +1,223 @@ +# Testing + +Developing lints for Clippy is a Test-Driven Development (TDD) process because +our first task before implementing any logic for a new lint is to write some test cases. + +## Develop Lints with Tests + +When we develop Clippy, we enter a complex and chaotic realm full of +programmatic issues, stylistic errors, illogical code and non-adherence to convention. +Tests are the first layer of order we can leverage to define when and where +we want a new lint to trigger or not. + +Moreover, writing tests first help Clippy developers to find a balance for +the first iteration of and further enhancements for a lint. +With test cases on our side, we will not have to worry about over-engineering +a lint on its first version nor missing out some obvious edge cases of the lint. +This approach empowers us to iteratively enhance each lint. + +## Clippy UI Tests + +We use **UI tests** for testing in Clippy. These UI tests check that the output +of Clippy is exactly as we expect it to be. Each test is just a plain Rust file +that contains the code we want to check. + +The output of Clippy is compared against a `.stderr` file. Note that you don't +have to create this file yourself. We'll get to generating the `.stderr` files +with the command [`cargo bless`](#cargo-bless) (seen later on). + +### Write Test Cases + +Let us now think about some tests for our imaginary `foo_functions` lint. We +start by opening the test file `tests/ui/foo_functions.rs` that was created by +`cargo dev new_lint`. + +Update the file with some examples to get started: + +```rust +#![warn(clippy::foo_functions)] // < Add this, so the lint is guaranteed to be enabled in this file + +// Impl methods +struct A; +impl A { + pub fn fo(&self) {} + pub fn foo(&self) {} //~ ERROR: function called "foo" + pub fn food(&self) {} +} + +// Default trait methods +trait B { + fn fo(&self) {} + fn foo(&self) {} //~ ERROR: function called "foo" + fn food(&self) {} +} + +// Plain functions +fn fo() {} +fn foo() {} //~ ERROR: function called "foo" +fn food() {} + +fn main() { + // We also don't want to lint method calls + foo(); + let a = A; + a.foo(); +} +``` + +Without actual lint logic to emit the lint when we see a `foo` function name, +this test will just pass, because no lint will be emitted. However, we can now +run the test with the following command: + +```sh +$ TESTNAME=foo_functions cargo uitest +``` + +Clippy will compile and it will conclude with an `ok` for the tests: + +``` +...Clippy warnings and test outputs... +test compile_test ... ok +test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.48s +``` + +This is normal. After all, we wrote a bunch of Rust code but we haven't really +implemented any logic for Clippy to detect `foo` functions and emit a lint. + +As we gradually implement our lint logic, we will keep running this UI test command. +Clippy will begin outputting information that allows us to check if the output is +turning into what we want it to be. + +### Example output + +As our `foo_functions` lint is tested, the output would look something like this: + +``` +failures: + +---- compile_test stdout ---- +normalized stderr: +error: function called "foo" + --> $DIR/foo_functions.rs:6:12 + | +LL | pub fn foo(&self) {} + | ^^^ + | + = note: `-D clippy::foo-functions` implied by `-D warnings` + +error: function called "foo" + --> $DIR/foo_functions.rs:13:8 + | +LL | fn foo(&self) {} + | ^^^ + +error: function called "foo" + --> $DIR/foo_functions.rs:19:4 + | +LL | fn foo() {} + | ^^^ + +error: aborting due to 3 previous errors +``` + +Note the *failures* label at the top of the fragment, we'll get rid of it +(saving this output) in the next section. + +> _Note:_ You can run multiple test files by specifying a comma separated list: +> `TESTNAME=foo_functions,bar_methods,baz_structs`. + +### `cargo bless` + +Once we are satisfied with the output, we need to run this command to +generate or update the `.stderr` file for our lint: + +```sh +$ TESTNAME=foo_functions cargo uibless +``` + +This writes the emitted lint suggestions and fixes to the `.stderr` file, with +the reason for the lint, suggested fixes, and line numbers, etc. + +Running `TESTNAME=foo_functions cargo uitest` should pass then. When we commit +our lint, we need to commit the generated `.stderr` files, too. + +In general, you should only commit files changed by `cargo bless` for the +specific lint you are creating/editing. + +> _Note:_ If the generated `.stderr`, and `.fixed` files are empty, +> they should be removed. + +## `toml` Tests + +Some lints can be configured through a `clippy.toml` file. Those configuration +values are tested in `tests/ui-toml`. + +To add a new test there, create a new directory and add the files: + +- `clippy.toml`: Put here the configuration value you want to test. +- `lint_name.rs`: A test file where you put the testing code, that should see a + different lint behavior according to the configuration set in the + `clippy.toml` file. + +The potential `.stderr` and `.fixed` files can again be generated with `cargo +bless`. + +## Cargo Lints + +The process of testing is different for Cargo lints in that now we are +interested in the `Cargo.toml` manifest file. In this case, we also need a +minimal crate associated with that manifest. Those tests are generated in +`tests/ui-cargo`. + +Imagine we have a new example lint that is named `foo_categories`, we can run: + +```sh +$ cargo dev new_lint --name=foo_categories --pass=late --category=cargo +``` + +After running `cargo dev new_lint` we will find by default two new crates, +each with its manifest file: + +* `tests/ui-cargo/foo_categories/fail/Cargo.toml`: this file should cause the + new lint to raise an error. +* `tests/ui-cargo/foo_categories/pass/Cargo.toml`: this file should not trigger + the lint. + +If you need more cases, you can copy one of those crates (under +`foo_categories`) and rename it. + +The process of generating the `.stderr` file is the same as for other lints +and prepending the `TESTNAME` variable to `cargo uitest` works for Cargo lints too. + +## Rustfix Tests + +If the lint you are working on is making use of structured suggestions, +[`rustfix`] will apply the suggestions from the lint to the test file code and +compare that to the contents of a `.fixed` file. + +Structured suggestions tell a user how to fix or re-write certain code that has +been linted with [`span_lint_and_sugg`]. + +Should `span_lint_and_sugg` be used to generate a suggestion, but not all +suggestions lead to valid code, you can use the `//@no-rustfix` comment on top +of the test file, to not run `rustfix` on that file. + +We'll talk about suggestions more in depth in a later chapter. + + +Use `cargo bless` to automatically generate the `.fixed` file after running +the tests. + +[`rustfix`]: https://github.com/rust-lang/rustfix +[`span_lint_and_sugg`]: https://doc.rust-lang.org/beta/nightly-rustc/clippy_utils/diagnostics/fn.span_lint_and_sugg.html + +## Testing Manually + +Manually testing against an example file can be useful if you have added some +`println!`s and the test suite output becomes unreadable. + +To try Clippy with your local modifications, run from the working copy root. + +```sh +$ cargo dev lint input.rs +```