bors
2a9e0831d6
Auto merge of #91393 - Julian-Wollersberger:lexer_optimization, r=petrochenkov
...
Optimize `rustc_lexer`
The `cursor.first()` method in `rustc_lexer` now calls the `chars.next()` method instead of `chars.nth_char(0)`.
This allows LLVM to optimize the code better. The biggest win is that `eat_while()` is now fully inlined and generates better assembly. This improves the lexer's performance by 35% in a micro-benchmark I made (Lexing all 18MB of code in the compiler directory). But lexing is only a small part of the overall compilation time, so I don't know how significant it is.
Big thanks to criterion and `cargo asm`.
2021-12-03 13:20:14 +00:00
Julian Wollersberger
1f147a2ed7
Replace nth_char(0) with next() in cursor.first()
...
and optimize the iterator returned by `tokenize().
This improves lexer performance by 35%
2021-12-01 19:14:10 +01:00
Esteban Kuber
38979a3ba1
udpate comment to be more accurate
2021-11-23 20:37:23 +00:00
Esteban Kuber
5a68abb094
Tokenize emoji as if they were valid indentifiers
...
In the lexer, consider emojis to be valid identifiers and reject
them later to avoid knock down parse errors.
2021-11-23 20:35:07 +00:00
Frank Steffahn
2396fad095
Fix more “a”/“an” typos
2021-08-22 17:27:18 +02:00
Ibraheem Ahmed
a397fdcc38
Remove ASCII fast path from rustc_lexer::{is_id_continue, is_id_start}
2021-07-26 20:17:28 -04:00
Mara Bos
0eeeebc990
Rename 'bad prefix' to 'unknown prefix'.
2021-06-26 23:11:14 +08:00
Mara Bos
6adce70a58
Improve comments for reserved prefixes.
...
Co-authored-by: Niko Matsakis <niko@alum.mit.edu>
2021-06-26 23:11:13 +08:00
lrh2000
8dee9bc8fc
Reserve prefixed identifiers and string literals (RFC 3101)
...
This commit denies any identifiers immediately followed by
one of three tokens `"`, `'` or `#`, which is stricter than
the requirements of RFC 3101 but may be necessary according
to the discussion at [Zulip].
[Zulip]: https://rust-lang.zulipchat.com/#narrow/stream/268952-edition-2021/topic/reserved.20prefixes/near/238470099
2021-06-26 23:09:43 +08:00
pierwill
0019ca9141
Fix outdated crate names in compiler docs
...
Changes `librustc_X` to `rustc_X`, only in documentation comments.
Plain code comments are left unchanged.
Also fix incorrect file paths.
2021-04-08 11:12:14 -05:00
Joshua Nelson
0ad3dce83a
Fix some clippy lints
2020-12-03 17:08:19 -05:00
Joshua Nelson
5339bd1ebe
Add back missing comments
2020-10-30 10:13:41 -04:00
Joshua Nelson
57c6ed0c07
Fix even more clippy warnings
2020-10-30 10:13:39 -04:00
Julian Wollersberger
bd49ded308
Noticed a potential bug in eat_while(): it doesn't account for number of UTF8 bytes.
...
Fixed it by inlining it in the two places where the count is used and simplified the logic there.
2020-10-09 11:12:54 +02:00
LingMan
fc20b7841c
Fix typo in rustc_lexer docs
...
Also add an Oxford comma while we're editing that line.
2020-09-21 05:43:39 +02:00
Vadim Petrochenkov
b1491eacfc
lexer: Tiny improvement to shebang detection
...
Lexer now discerns between regular comments and doc comments, so use that.
The change only affects the choice of reported errors.
2020-09-02 00:40:19 +03:00
mark
9e5f7d5631
mv compiler to compiler/
2020-08-30 18:45:07 +03:00
