add rustc_private as a proper language feature gate
At the moment, `rustc_private` as a (library) feature exists by
accident: `char::is_xid_start`, `char::is_xid_continue` methods in
libcore define it.
cc https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/How.20to.20declare.20new.20langauge.20feature.3F
I don't know if this is at all reasonable, but at least tests seem to pass locally. That probably means that we can remove/rename to something more resonable the feature in libcore in the next release?
move unescape module to rustc_lexer
It makes sense to keep the definition of escape sequences closer to the lexer itself, and it is also a bit of code that I would like to share with rust-analyzer.
r? @petrochenkov
rustc_typeck: improve diagnostics for _ const/static declarations
This continues https://github.com/rust-lang/rust/pull/62694 and adds type suggestions to const/static declarations with `_` type.
r? @eddyb
Use sharded maps for interning
Cuts down runtime from 5.5s to 3.8s for non-incremental `syntex_syntax` check builds with 16 threads / 8 cores.
r? @eddyb
Rollup of 14 pull requests
Successful merges:
- #62709 (Test that maplike FromIter satisfies uniqueness)
- #62713 (Stabilize <*mut _>::cast and <*const _>::cast)
- #62746 ( do not use assume_init in std::io)
- #62787 (Fix typo in src/libstd/net/udp.rs doc comment)
- #62788 (normalize use of backticks in compiler messages for libcore/ptr)
- #62799 (use const array repeat expressions for uninit_array)
- #62810 (normalize use of backticks in compiler messages for librustc_lint)
- #62812 (normalize use of backticks in compiler messages for librustc_metadata)
- #62832 (normalize use of backticks in compiler messages for librustc_incremental)
- #62845 (read: fix doc comment)
- #62853 (normalize use of backticks in compiler messages for librustc/hir)
- #62854 (Fix typo in Unicode character name)
- #62858 (Change wrong variable name.)
- #62870 (fix lexing of comments with many \r)
Failed merges:
r? @ghost
Fix typo in Unicode character name
There's a small typo in the Unicode character definitions: "Latin Epigraphic Letter Dideways" should be "Latin Epigraphic Letter Sideways I" (see [here](https://www.compart.com/en/unicode/U+A7F7)).
use const array repeat expressions for uninit_array
With a first implementation of https://github.com/rust-lang/rust/issues/49147 having landed, we can make this macro nicer and phase it out with the next bootstrap bump.
However, to make this work, we have to mark `MaybeUninit::uninit()` as promotable. I do feel uneasy about promoting stuff involving uninitialized memory, but OTOH no *operation* on `MaybeUninit` is promotable, so maybe this is okay?
r? @oli-obk @eddyb
Test that maplike FromIter satisfies uniqueness
This PR adds a simple assertion to the `HashMap` and `HashSet` tests to ensure that uniqueness is satisfied when `FromIter`ing. This is useful for people who want to test their custom type against the Map/Set interfaces since they'll copy the tests wholesale but possibly miss this bug (where _they_ = _me_).
ci: pin awscli dependencies
docutils 0.15, a dependency of awscli, broke our CI since it's not compatible with Python 2 due to a bug. This pins all the dependencies of awscli with docutils 0.14, to make sure this kind of regressions doesn't happen again.
r? @Mark-Simulacrum @alexcrichton
docutils 0.15, a dependency of awscli, broke our CI since it's not
compatible with Python 2 due to a bug. This pins all the dependencies of
awscli with docutils 0.14, to make sure this kind of regressions doesn't
happen again.
The essence of lexer
cc @eddyb
I would love to make a reusable library to lex rust code, which could be used by rustc, rust-analyzer, proc-macros, etc. This **draft** PR is my attempt at the API. Currently, the PR uses new lexer to lex comments and shebang, while using the old lexer for everything else. This should be enough to agree on the API though!
### High-level picture
An `rust_lexer` crate is introduced, with zero or minimal (for XID_Start and other unicode) dependencies. This crate basically exposes a single function: `next_token(&str) -> (TokenKind, usize)` which returns the first token of a non-empty string (`usize` is the length of the token). The main goal of the API is to be minimal. Non-strictly essential concerns, like string interning, are left to the clients.
### Finer Points
#### Iterator API
We probably should expose a convenience function `fn tokenize(&str) -> impl Iterator<Item = Token>`
EDIT: I've added `tokenize`
#### Error handling
The lexer itself provides only minimal amount of error detection and reporting. Additionally, it never fatal-errors and always produces some non-empty token. Examples of errors detected by the lexer:
* unterminated block comment
* unterminated string literals
Example of errors **not** detected by the lexer:
* invalid escape sequence in a string literal
* out of range integer literal
* bare `\r` in the doc comment.
The idea is that the clients are responsible for additional validation of tokens. This is the mode IDE operates in: you want to skip validation for library files, because you are not showing errors there anyway, and for user-code, you want to do a deep validation with quick fixes and suggestions, which is not really fit for the lexer itself.
In particular, in this PR unclosed `/*` comment is handled by the new lexer, bare `\r` and distinction between doc and non-doc comments is handled by the old lexer.
#### Performance
No attempt at performance measurement is made so far :) I think it is acceptable to regress perf here a bit in exchange for cleaner code, and I hope that regression wouldn't be too costly. In particular, because we validate tokens separately, we'll have to do one more pass for some of the tokens. I hope this is not a prohibitive cost. For example, for doc comments we already do two passes (lexing + interning), so adding a third one shouldn't be that much slower (and we also do an additional pass for utf-8 validation). And lexing is hopefully not a bottleneck. Note that for IDEs separate validation might actually improve performance, because we will be able to skip validation when, for example, computing completions.
Long term, I hope that this approach will allow for *better* performance. If we separate pure lexing, in the future we can code-gen super-optimizes state machine that walks utf-8 directly, instead of current manual char-by-char toil.
#### Cursor API
For implementation, I am going slightly unconventionally. Instead of defining a `Lexer` struct with a bunch of helper methods (`current`, `bump`) and a bunch of lexing methods (`lex_comment`, `lex_whitespace`), I define a `Cursor` struct which has only helpers, and define a top-level function with a `&mut Cursor` argument for each grammar production. I find this C-style more readable for parsers and lexers.
EDIT: swithced to a more conventional setup with lexing methods
So, what do folks think about this?
