6d7cfd4f1a
This commit adds even more pessimization to use the cached `TokenStream` inside of an AST node. As a reminder the `proc_macro` API requires taking an arbitrary AST node and transforming it back into a `TokenStream` to hand off to a procedural macro. Such functionality isn't actually implemented in rustc today, so the way `proc_macro` works today is that it stringifies an AST node and then reparses for a list of tokens. This strategy unfortunately loses all span information, so we try to avoid it whenever possible. Implemented in #43230 some AST nodes have a `TokenStream` cache representing the tokens they were originally parsed from. This `TokenStream` cache, however, has turned out to not always reflect the current state of the item when it's being tokenized. For example `#[cfg]` processing or macro expansion could modify the state of an item. Consequently we've seen a number of bugs (#48644 and #49846) related to using this stale cache. This commit tweaks the usage of the cached `TokenStream` to compare it to our lossy stringification of the token stream. If the tokens that make up the cache and the stringified token stream are the same then we return the cached version (which has correct span information). If they differ, however, then we will return the stringified version as the cache has been invalidated and we just haven't figured that out. Closes #48644 Closes #49846 |
||
|---|---|---|
| .. | ||
| auxiliary | ||
| proc-macro | ||
| ast_stmt_expr_attr.rs | ||
| compiler-calls.rs | ||
| create-dir-all-bare.rs | ||
| custom-derive-partial-eq.rs | ||
| derive-no-std-not-supported.rs | ||
| derive-totalsum-attr.rs | ||
| derive-totalsum.rs | ||
| deriving-encodable-decodable-box.rs | ||
| deriving-encodable-decodable-cell-refcell.rs | ||
| deriving-global.rs | ||
| dropck_tarena_sound_drop.rs | ||
| empty-struct-braces-derive.rs | ||
| extern-mod-syntax.rs | ||
| issue_24106.rs | ||
| issue-2804.rs | ||
| issue-4016.rs | ||
| issue-4036.rs | ||
| issue-11881.rs | ||
| issue-13560.rs | ||
| issue-14021.rs | ||
| issue-15149.rs | ||
| issue-15778-pass.rs | ||
| issue-15924.rs | ||
| issue-16723.rs | ||
| issue-16822.rs | ||
| issue-16992.rs | ||
| issue-18502.rs | ||
| issue-18763-quote-token-tree.rs | ||
| issue-24972.rs | ||
| issue-35829.rs | ||
| issue-40663.rs | ||
| linkage-visibility.rs | ||
| llvm-pass-plugin.rs | ||
| lto-syntax-extension.rs | ||
| macro-crate-does-hygiene-work.rs | ||
| macro-crate-multi-decorator-literals.rs | ||
| macro-crate-multi-decorator.rs | ||
| macro-crate.rs | ||
| macro-quote-cond.rs | ||
| macro-quote-test.rs | ||
| mbe_matching_test_macro.rs | ||
| myriad-closures.rs | ||
| outlive-expansion-phase.rs | ||
| plugin-args-1.rs | ||
| plugin-args-2.rs | ||
| plugin-args-3.rs | ||
| plugin-lib-ok-in-plugin.rs | ||
| plugin-plus-extern-crate.rs | ||
| pprust-expr-roundtrip.rs | ||
| proc_macro.rs | ||
| qquote.rs | ||
| quote-tokens.rs | ||
| quote-unused-sp-no-warning.rs | ||
| regions-mock-tcx.rs | ||
| rename-directory.rs | ||
| roman-numerals-macro.rs | ||
| rustc_encodable_hygiene.rs | ||
| stdio-from.rs | ||
| switch-stdout.rs | ||
| syntax-extension-with-dll-deps.rs | ||