Commit Graph

66 Commits

Author SHA1 Message Date
Matthias Krüger
2ea368e53c minor code cleanups 2022-12-12 19:49:53 +01:00
KaDiWa
9bc69925cb
compiler: remove unnecessary imports and qualified paths 2022-12-10 18:45:34 +01:00
klensy
ceee880b31 update cpufeatures, swap difference to dissimilar 2022-11-29 19:59:37 +03:00
bors
872631d0f0 Auto merge of #104507 - WaffleLapkin:asderefsyou, r=wesleywiser
Use `as_deref` in compiler (but only where it makes sense)

This simplifies some code :3

(there are some changes that are not exacly `as_deref`, but more like "clever `Option`/`Result` method use")
2022-11-24 00:17:35 +00:00
Maybe Waffle
94470f4efd Use as_deref in compiler (but only where it makes sense) 2022-11-16 21:58:58 +00:00
Nicholas Nethercote
358a603f11 Use token::Lit in ast::ExprKind::Lit.
Instead of `ast::Lit`.

Literal lowering now happens at two different times. Expression literals
are lowered when HIR is crated. Attribute literals are lowered during
parsing.

This commit changes the language very slightly. Some programs that used
to not compile now will compile. This is because some invalid literals
that are removed by `cfg` or attribute macros will no longer trigger
errors. See this comment for more details:
https://github.com/rust-lang/rust/pull/102944#issuecomment-1277476773
2022-11-16 09:41:28 +11:00
Dylan DPC
4b50fb3745
Rollup merge of #103919 - nnethercote:unescaping-cleanups, r=matklad
Unescaping cleanups

Some code improvements, and some error message improvements.

Best reviewed one commit at a time.

r? ````@matklad````
2022-11-09 19:21:22 +05:30
Nicholas Nethercote
d6c97a32b4 Simplify unescape_{char,byte}.
The `usize` isn't needed in the error case.
2022-11-08 15:59:19 +11:00
Nicholas Nethercote
dba6fc3ef5 Make underscore_literal_suffix a hard error.
It's been a warning for 5.5 years. Time to make it a hard error.

Closes #42326.
2022-11-07 10:00:36 +11:00
Nicholas Nethercote
43d21b535f Rename some result variables as res, for consistency. 2022-11-05 13:56:36 +11:00
Nicholas Nethercote
a838952239 Remove unescape_byte_literal.
It's easy to just use `unescape_literal` + `byte_from_char`.
2022-11-05 13:56:36 +11:00
Nicholas Nethercote
a203482d2a Inline and remove validate_int_literal.
It has a single callsite, and is fairly small. The `Float` match arm
already has base-specific checking inline, so this makes things more
consistent.
2022-11-04 14:24:41 +11:00
Nicholas Nethercote
a21c045897 Improve comments.
Remove a low-value comment, remove a duplicate comment, and correct a
third comment.
2022-11-04 14:24:41 +11:00
Nicholas Nethercote
7dbf2c0ed8 Make non-ASCII errors more consistent.
There are three kinds of "byte" literals: byte literals, byte string
literals, and raw byte string literals. None are allowed to have
non-ASCII chars in them.

Two `EscapeError` variants exist for when that constraint is violated.
- `NonAsciiCharInByte`: used for byte literals and byte string literals.
- `NonAsciiCharInByteString`: used for raw byte string literals.

As a result, the messages for raw byte string literals use different
wording, without good reason. Also, byte string literals are incorrectly
described as "byte constants" in some error messages.

This commit eliminates `NonAsciiCharInByteString` so the three cases are
handled similarly, and described correctly. The `mode` is enough to
distinguish them.

Note: Some existing error messages mention "byte constants" and some
mention "byte literals". I went with the latter here, because it's a
more correct name, as used by the Reference.
2022-11-04 14:23:40 +11:00
Nicholas Nethercote
34b32b0dac Use Mode less.
It's passed to numerous places where we just need an `is_byte` bool.
Passing the bool avoids the need for some assertions.

Also rename `is_bytes()` as `is_byte()`, to better match `Mode::Byte`,
`Mode::ByteStr`, and `Mode::RawByteStr`.
2022-11-03 15:58:19 +11:00
Nicholas Nethercote
84ca2c3bab Clarify range calculations.
There is some subtlety here.
2022-11-03 15:58:03 +11:00
Nicholas Nethercote
f32e6781b2 Rename some variables.
These have been bugging me for a while.

- `literal_text`: `src` is also used and is shorter and better.
- `first_char`: used even when "first" doesn't make sense; `c` is
  shorter and better.
- `curr`: `c` is shorter and better.
- `unescaped_char`: `result` is also used and is shorter and better.
- `second_char`: these have a single use and can be elided.
2022-11-03 11:58:11 +11:00
Tshepang Mbambo
b66f92197a rustc_lexer::TokenKind improve docs 2022-10-26 23:32:14 +02:00
reez12g
9a4c5abe45 Remove from compiler/ crates 2022-09-29 16:49:04 +09:00
bors
6201eabde8 Auto merge of #102302 - nnethercote:more-lexer-improvements, r=matklad
More lexer improvements

A follow-up to #99884.

r? `@matklad`
2022-09-28 08:14:04 +00:00
Nicholas Nethercote
d0a26acb2a Address review comments. 2022-09-28 11:15:23 +10:00
Nicholas Nethercote
94cb5e86ea Small cleanups in unescaping code.
- Rename `unescape_raw_str_or_raw_byte_str` as
  `unescape_raw_str_or_byte_str`, which is more accurate.
- Remove the unused `Mode::in_single_quotes` method.
- Make some assertions more precise, and add a missing one to
  `unescape_char_or_byte`.
- Change all the assertions to `debug_assert!`, because this code is
  reasonably hot, and the assertions aren't required for memory safety,
  and any violations are likely to be sufficiently obvious that normal
  tests will trigger them.
2022-09-28 08:31:24 +10:00
Nicholas Nethercote
c91c64708b Fix an incorrect comment.
If a `\x` escape occurs in a non-byte literals (e.g. char literal,
string literal), it must be <= 0xff.
2022-09-27 15:25:34 +10:00
Nicholas Nethercote
da84f0f4c3 Add rustc_lexer::TokenKind::Eof.
For alignment with `rust_ast::TokenKind::Eof`. Plus it's a bit faster,
due to less `Option` manipulation in `StringReader::next_token`.
2022-09-26 13:48:08 +10:00
Nicholas Nethercote
cc0022a363 Rename some things.
`Cursor` keeps track of the position within the current token. But it
uses confusing names that don't make it clear that the "length consumed"
is just within the current token.

This commit renames things to make this clearer.
2022-09-26 13:43:19 +10:00
Nicholas Nethercote
aa6bfaf04b Make rustc_lexer::cursor::Cursor public.
`Cursor` is currently hidden, and the main tokenization path uses
`rustc_lexer::first_token` which involves constructing a new `Cursor`
for every single token, which is weird. Also, `first_token` also can't
handle empty input, so callers have to check for that first.

This commit makes `Cursor` public, so `StringReader` can contain a
`Cursor`, which results in a simpler structure. The commit also changes
`StringReader::advance_token` so it returns an `Option<Token>`,
simplifying the the empty input case.
2022-09-26 13:36:35 +10:00
Takayuki Maeda
bdc865d8f7 remove unnecessary PartialOrd and Ord 2022-09-08 06:15:33 +09:00
5225225
09ea9f0a87 Add diagnostic translation lints to crates that don't emit them 2022-08-18 19:29:02 +01:00
Nicholas Nethercote
99f5c79d64 Shrink Token.
From 72 bytes to 12 bytes (on x86-64).

There are two parts to this:
- Changing various source code offsets from 64-bit to 32-bit. This is
  not a problem because the rest of rustc also uses 32-bit source code
  offsets. This means `Token` is no longer `Copy` but this causes no
  problems.
- Removing the `RawStrError` from `LiteralKind`. Raw string literal
  invalidity is now indicated by a `None` value within
  `RawStr`/`RawByteStr`, and the new `validate_raw_str` function can be
  used to re-lex an invalid raw string literal to get the `RawStrError`.

There is one very small change in behaviour. Previously, if a raw string
literal matched both the `InvalidStarter` and `TooManyHashes` cases,
the latter would override the former. This has now changed, because
`raw_double_quoted_string` now uses `?` and so returns immediately upon
detecting the `InvalidStarter` case. I think this is a slight
improvement to report the earlier-detected error, and it explains the
change in the `test_too_many_hashes` test.

The commit also removes a couple of comments that refer to #77629 and
say that the size of these types don't affect performance. These
comments are wrong, though the performance effect is small.
2022-08-01 08:53:04 +10:00
Nicholas Nethercote
b4fdf648ea Inline first_token.
Because it's tiny and hot.
2022-08-01 08:11:15 +10:00
Proloy Mishra
8c22b6bcac
fix typo in comment 2022-06-28 19:59:09 +05:30
Grisha Vartanyan
38e0ae590c Reduce max hash in raw strings from u16 to u8 2022-03-23 22:13:55 +01:00
Grisha
4e3dbb3c19
Add test for >65535 hashes in lexing raw string 2022-03-16 06:37:41 +01:00
Nicholas Nethercote
37d9ea745b Improve scan_escape.
`scan_escape` currently has a fast path (for when the first char isn't
'\\') and a slow path.

This commit changes `scan_escape` so it only handles the slow path, i.e.
the actual escaping code. The fast path is inlined into the two call
sites.

This change makes the code faster, because there is no function call
overhead on the fast path. (`scan_escape` is a big function and doesn't
get inlined.)

This change also improves readability, because it removes a bunch of
mode checks on the the fast paths.
2022-02-24 17:01:01 +11:00
bors
2a9e0831d6 Auto merge of #91393 - Julian-Wollersberger:lexer_optimization, r=petrochenkov
Optimize `rustc_lexer`

The `cursor.first()` method in `rustc_lexer` now calls the `chars.next()` method instead of `chars.nth_char(0)`.

This allows LLVM to optimize the code better. The biggest win is that `eat_while()` is now fully inlined and generates better assembly. This improves the lexer's performance by 35% in a micro-benchmark I made (Lexing all 18MB of code in the compiler directory). But lexing is only a small part of the overall compilation time, so I don't know how significant it is.

Big thanks to criterion and `cargo asm`.
2021-12-03 13:20:14 +00:00
Julian Wollersberger
1f147a2ed7 Replace nth_char(0) with next() in cursor.first()
and optimize the iterator returned by `tokenize().

This improves lexer performance by 35%
2021-12-01 19:14:10 +01:00
Esteban Kuber
38979a3ba1 udpate comment to be more accurate 2021-11-23 20:37:23 +00:00
Esteban Kuber
5a68abb094 Tokenize emoji as if they were valid indentifiers
In the lexer, consider emojis to be valid identifiers and reject
them later to avoid knock down parse errors.
2021-11-23 20:35:07 +00:00
Matthias Krüger
0a5640b55f use matches!() macro in more places 2021-11-06 16:13:14 +01:00
Matthias Krüger
4457014398 Revert "Auto merge of #89709 - clemenswasser:apply_clippy_suggestions_2, r=petrochenkov"
The PR had some unforseen perf regressions that are not as easy to find.
Revert the PR for now.

This reverts commit 6ae8912a3e, reversing
changes made to 86d6d2b738.
2021-10-15 11:28:23 +02:00
Clemens Wasser
71dd0b928b Apply clippy suggestions 2021-10-10 15:38:19 +02:00
Mark Rousskov
c746be2219 Migrate to 2021 2021-09-20 22:21:42 -04:00
Frank Steffahn
2396fad095 Fix more “a”/“an” typos 2021-08-22 17:27:18 +02:00
Anton Golov
07aacf53c5 Renamed variable str -> tail for clarity 2021-08-11 13:57:28 +02:00
Anton Golov
a03fbfe2ff Warn when an escaped newline skips multiple lines 2021-08-11 11:35:08 +02:00
bors
b53a93db2d Auto merge of #87535 - lf-:authors, r=Mark-Simulacrum
rfc3052 followup: Remove authors field from Cargo manifests

Since RFC 3052 soft deprecated the authors field, hiding it from
crates.io, docs.rs, and making Cargo not add it by default, and it is
not generally up to date/useful information for contributors, we may as well
remove it from crates in this repo.
2021-08-02 05:49:17 +00:00
Anton Golov
5d59b4412e Add warning when whitespace is not skipped after an escaped newline. 2021-07-30 16:26:39 +02:00
Jade
3cf820e17d rfc3052: Remove authors field from Cargo manifests
Since RFC 3052 soft deprecated the authors field anyway, hiding it from
crates.io, docs.rs, and making Cargo not add it by default, and it is
not generally up to date/useful information, we should remove it from
crates in this repo.
2021-07-29 14:56:05 -07:00
Ibraheem Ahmed
a397fdcc38
Remove ASCII fast path from rustc_lexer::{is_id_continue, is_id_start} 2021-07-26 20:17:28 -04:00
Mara Bos
0eeeebc990 Rename 'bad prefix' to 'unknown prefix'. 2021-06-26 23:11:14 +08:00