1238: Macro queries r=edwin0cheng a=matklad
In https://github.com/rust-analyzer/rust-analyzer/pull/1231, I've added aggressive clean up of `ast_id_to_node` query.
The result of this query is a `SyntaxTree`, and we don't want to retain syntax trees in memory unless absolutely necessary.
Moreover, `SyntaxTree` has identity equality semantics, meaning that we'll get a diffferent syntax tree for a file after every reparse. That means that `ast_id_to_node` query should not genereally be used in HIR, unless it is behind some kind of salsa firewall, like the `raw` module of name resoulution.
However, that PR resulted in the abysmal performance: turns out we were using ast_id_to_node quite heavily in hir when expanding macros!
So this PR installs the more incremental-friendly query structure:
* converting source to token tree is now a query; changing source without affecting token-trees will now preserve macro expansions
* expand macro (tt -> tt) is now a query as well, so we cache macro expansions *before* parsing them into item lists or expressions, which is nice: we can cache expansion without knowing the calling context!
r? @edwin0cheng
Co-authored-by: Aleksey Kladov <aleksey.kladov@gmail.com>
Currently, when expanding macros, we look at the source code
directly (we invoke ast_id_to_node query via to_node method).
This is less then ideal, because it make us re-expand macros after
every source change.
This commit establishes a salsa-firewall: a query to get macro call's
token tree. Unlike the syntax tree, token tree changes only if we
actually modify the macro itself.
1237: Improve $ Handling in mbe parser r=matklad a=edwin0cheng
This PR improve the $ handling in mbe parser. In some rare case, the `$` may not be following an `ident` or a `Subtree`. ( For example, a macro_rules inside a macro rules).
Related issue: #1236
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1230: Desugar doc comments to `#[doc = "...."]` attributes in `syntax node` to tt conversion r=matklad a=edwin0cheng
As discussed in [Zulip](https://rust-lang.zulipchat.com/#narrow/stream/185405-t-compiler.2Fwg-rls-2.2E0/topic/MBE.20discussion/near/164446835), this PR desugar doc comments to `#[doc = "...."]` in `syntax node` to tt conversion.
Note that after this PR, all obvious mbe bugs in dogfooding are fixed. (i.e. No parsing or expanding mbe error in `env RUST_LOG=ra_hir=WARN target\release\ra_cli.exe analysis-stats`) 🎉
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1229: Mark unused mbe variable as `Binding::Empty` r=matklad a=edwin0cheng
This PR fixes a regression bug in In #1228, which incorrect expand an empty binding :
* Introduce a new Binding type `Binding::Empty`.
* Mark all unused binding variables are empty and error out in expansion instead of just by passing.
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1228: Add inner macro checker in mbe r=matklad a=edwin0cheng
This PR do the following things:
* Add an inner macro checker for allowing defining mbe in mbe. (It is a adhoc solution, we could eliminate it after we have a better mbe parser)
* Move all tests to an tests modules
* Filter empty tree while expanding mbe.
This is the final PR extracting from #1219
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1227: Add `default_type` method in `TypeParam` Node r=matklad a=edwin0cheng
This PR add a `default_type` method in `TypeParam` Node which allow future PR to handle #1099 case.
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1226: Fix Bug in mbe on mismatching bindings count r=matklad a=edwin0cheng
This PR do the followings things:
1. Fixed a bug happened in following code by inserting empty bindings before a nested bindings:
```
macro_rules foo! {
($( ($($a:ident)* => $($b:ident))*))* => { ... }
}
foo!( ( => a) (b => ) )
```
2. Fixed a bug which forget to restore `var_expanded`.
3. Some cleanup
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1224: Remove unused multchar puncts code and add space between puncts r=matklad a=edwin0cheng
After #1213 , parser only need single char punct, this PR do the following things:
* Remove code which handles multi char puncts
* Remove code which handle traversal backward in `SubtreeSource` , because we cached the result in #1195
* Add space between two consecutive puncts while `tt` to `SyntaxNode` conversion .
Note that the spaces should only be added if both puncts are not delimiters.
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1222: Skip Dollars when bump raw token r=matklad a=edwin0cheng
This PR fixed a bug while parsing token_tree, it should skip all L_DOLLAR AND R_DOLLAR.
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1220: Add macro pat parsing r=matklad a=edwin0cheng
This PR add support to parsing macro call in pattern , e.g :
```
let m!(x) = 0;
```
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1217: Sidestep two-phase borrow violation r=matklad a=pnkfelix
Sidestep two-phase borrow violation signaled by mutable_borrow_reservation_conflict.
See rust-lang/rust#59159 for further information/discussion.
Co-authored-by: Felix S. Klock II <pnkfelix@pnkfx.org>
1213: Make lexer produce only single character puncts r=matklad a=edwin0cheng
As discussed in Zulip, this PR change `lexer` to produce only single char punct.
* Remove producing `DOTDOTDOT, DOTDOTEQ, DOTDOT, COLONCOLON, EQEQ, FAT_ARROW, NEQ, THIN_ARROW` in lexer.
* Add required code in parser to make sure everythings works fine.
* Change some tests (Mainly because the `ast::token_tree` is different)
Note: i think the use of `COLON` in rust is too overloaded :)
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1215: Fix hover on the beginning of a nested expression r=matklad a=flodiebold
E.g. in
```
let foo = 1u32;
if true {
<|>foo;
}
```
the hover shows `()`, the type of the whole if expression, instead of the more
sensible `u32`. The reason for this was that the search for an expression was
slightly left-biased: When on the edge between two tokens, it first looked at
all ancestors of the left token and then of the right token. Instead merge the
ancestors in ascending order, so that we get the smaller of the two possible
expressions.
This might seem rather inconsequential, but emacs-lsp's sideline requests hover for the beginning of each word in the current line, so it tends to hit this a lot 😄 Actually, as I think about this now, another solution might be just making hover right-biased, since if I'm hovering a certain character I really mean that character and not the one to its left...
Co-authored-by: Florian Diebold <flodiebold@gmail.com>
E.g. in
```
let foo = 1u32;
if true {
<|>foo;
}
```
the hover shows `()`, the type of the whole if expression, instead of the more
sensible `u32`. The reason for this was that the search for an expression was
slightly left-biased: When on the edge between two tokens, it first looked at
all ancestors of the left token and then of the right token. Instead merge the
ancestors in ascending order, so that we get the smaller of the two possible
expressions.
