1336: Refactor SubtreeSource r=matklad a=edwin0cheng
This PR simplify `SubtreeSource` by removing `SubtreeWalk` and `Querier` and only walk through the top level `TokenTree` when collecting token from source, by comparing two cursors directly.
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
Very simple approach: For each identifier, set the hash of the range
where it's defined as its 'id' and use it in the VSCode extension to
generate unique colors.
Thus, the generated colors are per-file. They are also quite fragile,
and I'm not entirely sure why. Looks like we need to make sure the
same ranges aren't overwritten by a later request?
1328: Change TokenSource to iteration based r=matklad a=edwin0cheng
This PR change the `TokenSource` trait from random access to be an iteration based trait:
```rust
/// `TokenSource` abstracts the source of the tokens parser operates one.
///
/// Hopefully this will allow us to treat text and token trees in the same way!
pub trait TokenSource {
fn current(&self) -> Token;
/// Lookahead n token
fn lookahead_nth(&self, n: usize) -> Token;
/// bump cursor to next token
fn bump(&mut self);
/// Is the current token a specified keyword?
fn is_keyword(&self, kw: &str) -> bool;
}
/// `TokenCursor` abstracts the cursor of `TokenSource` operates one.
#[derive(Debug, Copy, Clone, Eq, PartialEq)]
pub struct Token {
/// What is the current token?
pub kind: SyntaxKind,
/// Is the current token joined to the next one (`> >` vs `>>`).
pub is_jointed_to_next: bool,
}
```
Note that the refactoring based on this new trait will be separated to incoming PRs
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1316: Simplify code model r=matklad a=matklad
* remove references from types which are now id-based
* remove api/impl separation, as the impl is a tiny fraction of API anyway
Co-authored-by: Aleksey Kladov <aleksey.kladov@gmail.com>
