The essence of lexer
cc @eddyb
I would love to make a reusable library to lex rust code, which could be used by rustc, rust-analyzer, proc-macros, etc. This **draft** PR is my attempt at the API. Currently, the PR uses new lexer to lex comments and shebang, while using the old lexer for everything else. This should be enough to agree on the API though!
### High-level picture
An `rust_lexer` crate is introduced, with zero or minimal (for XID_Start and other unicode) dependencies. This crate basically exposes a single function: `next_token(&str) -> (TokenKind, usize)` which returns the first token of a non-empty string (`usize` is the length of the token). The main goal of the API is to be minimal. Non-strictly essential concerns, like string interning, are left to the clients.
### Finer Points
#### Iterator API
We probably should expose a convenience function `fn tokenize(&str) -> impl Iterator<Item = Token>`
EDIT: I've added `tokenize`
#### Error handling
The lexer itself provides only minimal amount of error detection and reporting. Additionally, it never fatal-errors and always produces some non-empty token. Examples of errors detected by the lexer:
* unterminated block comment
* unterminated string literals
Example of errors **not** detected by the lexer:
* invalid escape sequence in a string literal
* out of range integer literal
* bare `\r` in the doc comment.
The idea is that the clients are responsible for additional validation of tokens. This is the mode IDE operates in: you want to skip validation for library files, because you are not showing errors there anyway, and for user-code, you want to do a deep validation with quick fixes and suggestions, which is not really fit for the lexer itself.
In particular, in this PR unclosed `/*` comment is handled by the new lexer, bare `\r` and distinction between doc and non-doc comments is handled by the old lexer.
#### Performance
No attempt at performance measurement is made so far :) I think it is acceptable to regress perf here a bit in exchange for cleaner code, and I hope that regression wouldn't be too costly. In particular, because we validate tokens separately, we'll have to do one more pass for some of the tokens. I hope this is not a prohibitive cost. For example, for doc comments we already do two passes (lexing + interning), so adding a third one shouldn't be that much slower (and we also do an additional pass for utf-8 validation). And lexing is hopefully not a bottleneck. Note that for IDEs separate validation might actually improve performance, because we will be able to skip validation when, for example, computing completions.
Long term, I hope that this approach will allow for *better* performance. If we separate pure lexing, in the future we can code-gen super-optimizes state machine that walks utf-8 directly, instead of current manual char-by-char toil.
#### Cursor API
For implementation, I am going slightly unconventionally. Instead of defining a `Lexer` struct with a bunch of helper methods (`current`, `bump`) and a bunch of lexing methods (`lex_comment`, `lex_whitespace`), I define a `Cursor` struct which has only helpers, and define a top-level function with a `&mut Cursor` argument for each grammar production. I find this C-style more readable for parsers and lexers.
EDIT: swithced to a more conventional setup with lexing methods
So, what do folks think about this?
The idea here is to make a reusable library out of the existing
rust-lexer, by separating out pure lexing and rustc-specific concerns,
like spans, error reporting an interning.
So, rustc_lexer operates directly on `&str`, produces simple tokens
which are a pair of type-tag and a bit of original text, and does not
report errors, instead storing them as flags on the token.
Specific error for positional args after named args in `format!()`
When writing positional arguments after named arguments in the
`format!()` and `println!()` macros, provide a targeted diagnostic.
Follow up to https://github.com/rust-lang/rust/pull/57522/files#r247278885
Add meta-variable checks in macro definitions
This is an implementation of #61053. It is not sound (some errors are not reported) and not complete (reports may not be actual errors). This is due to the possibility to define macros in macros in indirect ways. See module documentation of `macro_check` for more details.
What remains to be done:
- [x] Migrate from an error to an allow-by-default lint.
- [x] Add more comments in particular for the handling of nested macros.
- [x] Add more tests if needed.
- [x] Try to avoid cloning too much (one idea is to use lists on the stack).
- [ ] Run crater with deny-by-default lint (measure rate of false positives).
- [ ] Remove extra commit for deny-by-default lint
- [x] Create a PR to remove the old `question_mark_macro_sep` lint #62160
libsyntax: Rename `Mark` into `ExpnId`
"`Mark`" is an ID that identifies both a macro invocation (`foo!()`), and expansion process, and expansion result of that macro invocation.
The problem is that it's pretty hard to infer that from its name.
This PR renames it into `ExpnId` reflecting its meaning in most contexts.
(The contexts where it's meaning is closer to "macro invocation ID" are rarer.)
I've kept "mark" in the names of functions that add or remove marks to/from syntactic contexts, those marks are not just expansion IDs, but something more complex.
