Shrink Token.

From 72 bytes to 12 bytes (on x86-64).

There are two parts to this:
- Changing various source code offsets from 64-bit to 32-bit. This is
  not a problem because the rest of rustc also uses 32-bit source code
  offsets. This means `Token` is no longer `Copy` but this causes no
  problems.
- Removing the `RawStrError` from `LiteralKind`. Raw string literal
  invalidity is now indicated by a `None` value within
  `RawStr`/`RawByteStr`, and the new `validate_raw_str` function can be
  used to re-lex an invalid raw string literal to get the `RawStrError`.

There is one very small change in behaviour. Previously, if a raw string
literal matched both the `InvalidStarter` and `TooManyHashes` cases,
the latter would override the former. This has now changed, because
`raw_double_quoted_string` now uses `?` and so returns immediately upon
detecting the `InvalidStarter` case. I think this is a slight
improvement to report the earlier-detected error, and it explains the
change in the `test_too_many_hashes` test.

The commit also removes a couple of comments that refer to #77629 and
say that the size of these types don't affect performance. These
comments are wrong, though the performance effect is small.
This commit is contained in:
Nicholas Nethercote 2022-07-27 13:59:30 +10:00
parent 3641b0ca30
commit 09f9acea0a
3 changed files with 4 additions and 4 deletions

View File

@ -1112,7 +1112,7 @@ fn span_contains_cfg(cx: &LateContext<'_>, s: Span) -> bool {
let mut pos = 0usize;
let mut iter = tokenize(&snip).map(|t| {
let start = pos;
pos += t.len;
pos += t.len as usize;
(t.kind, start..pos)
});

View File

@ -345,7 +345,7 @@ fn text_has_safety_comment(src: &str, line_starts: &[BytePos], offset: usize) ->
if line.starts_with("/*") {
let src = src[line_start..line_starts.last().unwrap().to_usize() - offset].trim_start();
let mut tokens = tokenize(src);
return src[..tokens.next().unwrap().len]
return src[..tokens.next().unwrap().len as usize]
.to_ascii_uppercase()
.contains("SAFETY:")
&& tokens.all(|t| t.kind == TokenKind::Whitespace);

View File

@ -141,7 +141,7 @@ fn eq_block(&mut self, left: &Block<'_>, right: &Block<'_>) -> bool {
let mut left_pos = 0;
let left = tokenize(&left)
.map(|t| {
let end = left_pos + t.len;
let end = left_pos + t.len as usize;
let s = &left[left_pos..end];
left_pos = end;
(t, s)
@ -156,7 +156,7 @@ fn eq_block(&mut self, left: &Block<'_>, right: &Block<'_>) -> bool {
let mut right_pos = 0;
let right = tokenize(&right)
.map(|t| {
let end = right_pos + t.len;
let end = right_pos + t.len as usize;
let s = &right[right_pos..end];
right_pos = end;
(t, s)