1336: Refactor SubtreeSource r=matklad a=edwin0cheng
This PR simplify `SubtreeSource` by removing `SubtreeWalk` and `Querier` and only walk through the top level `TokenTree` when collecting token from source, by comparing two cursors directly.
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
Very simple approach: For each identifier, set the hash of the range
where it's defined as its 'id' and use it in the VSCode extension to
generate unique colors.
Thus, the generated colors are per-file. They are also quite fragile,
and I'm not entirely sure why. Looks like we need to make sure the
same ranges aren't overwritten by a later request?
1328: Change TokenSource to iteration based r=matklad a=edwin0cheng
This PR change the `TokenSource` trait from random access to be an iteration based trait:
```rust
/// `TokenSource` abstracts the source of the tokens parser operates one.
///
/// Hopefully this will allow us to treat text and token trees in the same way!
pub trait TokenSource {
fn current(&self) -> Token;
/// Lookahead n token
fn lookahead_nth(&self, n: usize) -> Token;
/// bump cursor to next token
fn bump(&mut self);
/// Is the current token a specified keyword?
fn is_keyword(&self, kw: &str) -> bool;
}
/// `TokenCursor` abstracts the cursor of `TokenSource` operates one.
#[derive(Debug, Copy, Clone, Eq, PartialEq)]
pub struct Token {
/// What is the current token?
pub kind: SyntaxKind,
/// Is the current token joined to the next one (`> >` vs `>>`).
pub is_jointed_to_next: bool,
}
```
Note that the refactoring based on this new trait will be separated to incoming PRs
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1316: Simplify code model r=matklad a=matklad
* remove references from types which are now id-based
* remove api/impl separation, as the impl is a tiny fraction of API anyway
Co-authored-by: Aleksey Kladov <aleksey.kladov@gmail.com>
1290: Add Union to code_model r=matklad a=matklad
@flodiebold I am conflicted about two possible implementation approaches:
* we can add a separate `struct Union` to code model
* we can add `fn is_union(&self)` to existing `Struct`
This PR goes with the former approach, because it seems like Unions are sufficiently different in semantics to warrant a separate types. Which is in contrast to Syntax Tree, where both structs and unions share the same node kind, because their syntax is the same.
What would be the right thing to do here?
Co-authored-by: Aleksey Kladov <aleksey.kladov@gmail.com>
1312: Introduce TokenBuffer r=matklad a=edwin0cheng
As discussed in Zulip, this PR Introduce `TokenBuffer` , a safe version of `syn` crate `TokenBuffer` which support cursor based traversal of `tt::TokenTree`. This is the basis of incoming refactoring of `TokenSource` iterator based API.
This PR do the following things:
* Add TokenBuffer in `ra_tt` crate.
* Try to use this new API to refactor the `SubtreeSource` to prove it usage.
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1281: Move arm cond to match guard r=matklad a=unrealhoang
I did split the rename to another commit, yet Github UI still show entirely new file change. Please review using commits.
Co-authored-by: Unreal Hoang <unrealhoang@gmail.com>
1299: Use ThemeColor and add support for light themes r=matklad a=lnicola
Part of #1294.
- switch to `ThemeColor`
- add light and high contrast theme definitions
- highlight control flow keywords and `unsafe`
Co-authored-by: Laurențiu Nicola <lnicola@dend.ro>
1287: Add support of matching literal in mbe r=matklad a=edwin0cheng
This PR adds support of matching literal in mbe , which used in our `T` macro :
```rust
macro_rules! foo {
('(') => {
fn foo() {}
}
}
```
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1286: Add infer for generic default type r=flodiebold a=edwin0cheng
This PR add infer support for generic default type:
```
struct Gen<T=u32> {
val: T
}
```
* add the (unresolved) defaults from the definition to GenericParams
* add a query generic_defaults that resolves those defaults to types and returns a Substs
* add the missing type in `substs_from_path_segment`
* add tests
based on the idea in this [comment](https://github.com/rust-analyzer/rust-analyzer/issues/1099#issuecomment-484206279)
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1272: Error out non single root token tree conversion r=matklad a=edwin0cheng
This PR add a check to prevent non single root token tree conversion between token trees and syntax tree.
It should prevent the assert produced in #1267.
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
Reducing it to 2 was just a failed attempt to see whether that would help fix
some slow cases; in fact, it can create new slow cases by replacing concrete
types by variables.
For Send/Sync/Sized, we don't handle auto traits correctly yet and because they
have a lot of impls, they can easily lead to slowdowns. In the case of
Fn/FnMut/FnOnce, we don't parse the special Fn notation correctly yet and don't
handle closures yet, so we are very unlikely to find an impl.
This is slightly hacky, but maybe more elegant than alternative solutions: We
just use a hardcoded Chalk trait ID which we special-case to have no impls.
1251: Chalk integration improvements r=matklad a=flodiebold
A few improvements that came up while working on where clause support:
- turn `implements` into a query again to improve performance
- allow skipping to a specific function with `analysis-stats`, e.g. `ra_cli analysis-stats --only world_symbols`
- deduplicate impls in impls_for_trait -- previously many impls e.g. from std where repeated many times, this should help performance as well...
- add a `HirDisplay` implementation for TraitRef (not used here anywhere, but useful for debugging)
Co-authored-by: Florian Diebold <flodiebold@gmail.com>
1208: [WIP] Goto for Macro's r=matklad a=Lapz
Adds goto definition for macros. Currently only works for macros in the current crate ~~otherwise it panics~~. Proper macro resolution needs to be added for it to resolve macros in other crates.
Todo
- [X] Allow goto from macro calls
- [X] Fix panics
- [x] Add tests
Co-authored-by: Lenard Pratt <l3np27@gmail.com>
1216: Basic Chalk integration r=matklad a=flodiebold
This replaces the ad-hoc `implements` check by Chalk. It doesn't yet any new functionality (e.g. where clauses aren't passed to Chalk yet). The tests that exist actually work, but it needs some refactoring, currently crashes when running analysis on the RA repo, and depends on rust-lang/chalk#216 which isn't merged yet 😄
The main work here is converting stuff back and forth and providing Chalk with the information it needs, and the canonicalization logic. Since canonicalization depends a lot on the inference table, I don't think we can currently reuse the logic from Chalk, so we need to implement it ourselves; it's not actually that complicated anyway ;) I realized that we need a `Ty::Bound` variant separate from `Ty::Param` -- these are two different things, and I think type parameters inside a function actually need to be represented in Chalk as `Placeholder` types.
~~Currently this crashes in the 'real' world because we don't yet do canonicalization when filtering method candidates. Proper canonicalization needs the inference table (to collapse different inference variables that have already been unified), but we need to be able to call the method candidate selection from the completion code... So I'm currently thinking how to best handle that 😄~~
Co-authored-by: Florian Diebold <flodiebold@gmail.com>
1238: Macro queries r=edwin0cheng a=matklad
In https://github.com/rust-analyzer/rust-analyzer/pull/1231, I've added aggressive clean up of `ast_id_to_node` query.
The result of this query is a `SyntaxTree`, and we don't want to retain syntax trees in memory unless absolutely necessary.
Moreover, `SyntaxTree` has identity equality semantics, meaning that we'll get a diffferent syntax tree for a file after every reparse. That means that `ast_id_to_node` query should not genereally be used in HIR, unless it is behind some kind of salsa firewall, like the `raw` module of name resoulution.
However, that PR resulted in the abysmal performance: turns out we were using ast_id_to_node quite heavily in hir when expanding macros!
So this PR installs the more incremental-friendly query structure:
* converting source to token tree is now a query; changing source without affecting token-trees will now preserve macro expansions
* expand macro (tt -> tt) is now a query as well, so we cache macro expansions *before* parsing them into item lists or expressions, which is nice: we can cache expansion without knowing the calling context!
r? @edwin0cheng
Co-authored-by: Aleksey Kladov <aleksey.kladov@gmail.com>
Currently, when expanding macros, we look at the source code
directly (we invoke ast_id_to_node query via to_node method).
This is less then ideal, because it make us re-expand macros after
every source change.
This commit establishes a salsa-firewall: a query to get macro call's
token tree. Unlike the syntax tree, token tree changes only if we
actually modify the macro itself.
1230: Desugar doc comments to `#[doc = "...."]` attributes in `syntax node` to tt conversion r=matklad a=edwin0cheng
As discussed in [Zulip](https://rust-lang.zulipchat.com/#narrow/stream/185405-t-compiler.2Fwg-rls-2.2E0/topic/MBE.20discussion/near/164446835), this PR desugar doc comments to `#[doc = "...."]` in `syntax node` to tt conversion.
Note that after this PR, all obvious mbe bugs in dogfooding are fixed. (i.e. No parsing or expanding mbe error in `env RUST_LOG=ra_hir=WARN target\release\ra_cli.exe analysis-stats`) 🎉
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1227: Add `default_type` method in `TypeParam` Node r=matklad a=edwin0cheng
This PR add a `default_type` method in `TypeParam` Node which allow future PR to handle #1099 case.
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1222: Skip Dollars when bump raw token r=matklad a=edwin0cheng
This PR fixed a bug while parsing token_tree, it should skip all L_DOLLAR AND R_DOLLAR.
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1220: Add macro pat parsing r=matklad a=edwin0cheng
This PR add support to parsing macro call in pattern , e.g :
```
let m!(x) = 0;
```
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1213: Make lexer produce only single character puncts r=matklad a=edwin0cheng
As discussed in Zulip, this PR change `lexer` to produce only single char punct.
* Remove producing `DOTDOTDOT, DOTDOTEQ, DOTDOT, COLONCOLON, EQEQ, FAT_ARROW, NEQ, THIN_ARROW` in lexer.
* Add required code in parser to make sure everythings works fine.
* Change some tests (Mainly because the `ast::token_tree` is different)
Note: i think the use of `COLON` in rust is too overloaded :)
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
E.g. in
```
let foo = 1u32;
if true {
<|>foo;
}
```
the hover shows `()`, the type of the whole if expression, instead of the more
sensible `u32`. The reason for this was that the search for an expression was
slightly left-biased: When on the edge between two tokens, it first looked at
all ancestors of the left token and then of the right token. Instead merge the
ancestors in ascending order, so that we get the smaller of the two possible
expressions.
1200: Allows searching for case-equivalent symbols (fixes#1151) r=matklad a=jrvidal
I couldn't find a nice, functional way of calculating the ranges in one pass so I resorted to a plain old `for` loop.
Co-authored-by: Roberto Vidal <vidal.roberto.j@gmail.com>
1147: Handle macros in type checking / HIR r=matklad a=Lapz
An other attempt at #1102. I will need to flatten the nested if statements and im also not sure if the way that i get the resolver and module will always work
Co-authored-by: Lenard Pratt <l3np27@gmail.com>
We really shouldn't be looking at the identifier at point. Instead,
all filtering and sorting should be implemented at the layer above.
This layer should probably be home for auto-import completions as
well, but, since that is not yet implemented, let's just stick this
into complete_scope.
1193: Add a test for #1178 case r=edwin0cheng a=edwin0cheng
A little PR to add a test case for #1178
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1184: Start structured editing API r=matklad a=matklad
I think I finally understand how to provide nice, mutable structured editing API on top of red-green trees.
The problem I am trying to solve is that any modification to a particular `SyntaxNode` returns an independent new file. So, if you are editing a struct literal, and add a field, you get back a SourceFile, and you have to find the struct literal inside it yourself! This happens because our trees are immutable, but have parent pointers.
The main idea here is to introduce `AstEditor<T>` type, which abstracts away that API. So, you create an `AstEditor` for node you want to edit and call various `&mut` taking methods on it. Internally, `AstEditor` stores both the original node and the current node. All edits are applied to the current node, which is replaced by the corresponding node in the new file. In the end, `AstEditor` computes a text edit between old and new nodes.
Note that this also should sole a problem when you create an anchor pointing to a subnode and mutate the parent node, invalidating anchor. Because mutation needs `&mut`, all anchors must be killed before modification.
Co-authored-by: Aleksey Kladov <aleksey.kladov@gmail.com>
1175: Fix bugs and add error log about macro expansion r=matklad a=edwin0cheng
This PR fixed / add following things:
* Add a fused count which stop recursion of macro expansion in name resolution.
* Add some logs when macro expansion fails
* Add `$crate` meta variable support in mbe, which create a `$crate` ident token in token tree.
* Fixed matching a `$REPEAT` pattern inside a subtree, e.g. `(fn $name:ident {$($i:ident)*} ) => {...}`
* Remove composite-able punct token in syntax node to token conversion.
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1154: Initial support for lang items (and str completion) r=flodiebold a=marcogroppo
This PR adds partial support for lang items.
For now, the only supported lang items are the ones that target an impl block.
Lang items are now resolved during type inference - this means that `str` completion now works.
Fixes#1139.
(thanks Florian Diebold for the help!)
Co-authored-by: Marco Groppo <marco.groppo@gmail.com>
1148: Add token_tree_to_xxx functions r=matklad a=edwin0cheng
<del>As discus in PR #1147 , this PR added a `mbe::MacroKind` .
Currently only 2 kind of macro are supported, `SourceFile` and `Block`.</del>
Added following functions for `tt::TokenTree` and `ast::Node` conversion:
* token_tree_to_expr
* token_tree_to_pat
* token_tree_to_ty
* token_tree_to_macro_stmts
* token_tree_to_macro_items
And added two new syntax kind:
* MACRO_ITEMS
* MACRO_STMTS
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1145: Fix#1099, plug self type type parameters in infer_path_expr r=flodiebold a=edwin0cheng
As discussed in #1099, this PR try to "plug" a `self type` type parameters in `infer_path_expr`.
All the cases in 1099 was fixed and tested.
And luckily, this PR fixed bug #1030 again and make the test output correct.
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1153: "Restart server" command r=jrvidal a=jrvidal
The only tricky aspect is that fact that once the `exit` command has been received, we no longer need to join on the reader thread.
Also, I think `terminateProcesses.sh` was not working properly. In fact, the very same script from the vscode language server implementation is not working either! It's because of that I noticed the reader thread issue 😮
Co-authored-by: Roberto Vidal <vidal.roberto.j@gmail.com>
This fixes the order in which candidates are chosen a bit (not completely
though, as the ignored test demonstrates), and makes autoderef work with trait
methods. As a side effect, this also makes completion of trait methods work :)
1137: Adds support for multiple editor workspaces on initialization r=matklad a=jrvidal
OK, so this "simple hack" turned out to be way more contrived than I expected 😂
### What works
This patch only handles multi-folder editor workspaces _on initialization_.
* I've found that modifying the layout of a workspace in VSCode just reloads the extension, so this hack should be enough for now.
* Not sure about how emacs-lsp behaves, but we fallback gracefully to the mono-folder workspace, so it should be fine.
### What doesn't work
* [x] `cargo watch` can only watch a single root folder with a `Cargo.toml`. I've left this part untouched but we could either warn that it's not supported or launch _multiple_ `cargo-watch` processes.
* [x] The `rust-analyzer/runnables` command is not functional, since we don't send the correct `cwd`.
* [x] Should we add some happy path test to `heavy_tests`?
* [ ] Going from a single `root` to multiple `roots` leaves us with a couple of `n * m` loops that smell a bit. The number of folders in the editor workspace is probably low though.
Co-authored-by: Roberto Vidal <vidal.roberto.j@gmail.com>
1138: Add L_DOLLAR and R_DOLLAR r=matklad a=edwin0cheng
As discussion in issue https://github.com/rust-analyzer/rust-analyzer/issues/1132 and PR #1125 , this PR add 2 `Syntax::Kind` : `L_DOLLAR` and `R_DOLLAR` for representing `Delimiter::None` in mbe and proc_marco. By design, It should not affect the final syntax tree, and will be discard in `TreeSink`.
My original idea is handling these 2 tokens case by case, but i found that they will appear in every place in the parser (imagine `tt` matcher). So this PR only handle it in `Parser::do_bump` and `Parser::start`, although It will not fix the `expr` matcher executing order problem in original idea.
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
- make it possible to get parent trait from method
- add 'obligation' machinery for checking that a type implements a
trait (and inferring facts about type variables from that)
- handle type parameters of traits (to a certain degree)
- improve the hacky implements check to cover enough cases to exercise the
handling of traits with type parameters
- basic canonicalization (will probably also be done by Chalk)
