1328: Change TokenSource to iteration based r=matklad a=edwin0cheng
This PR change the `TokenSource` trait from random access to be an iteration based trait:
```rust
/// `TokenSource` abstracts the source of the tokens parser operates one.
///
/// Hopefully this will allow us to treat text and token trees in the same way!
pub trait TokenSource {
fn current(&self) -> Token;
/// Lookahead n token
fn lookahead_nth(&self, n: usize) -> Token;
/// bump cursor to next token
fn bump(&mut self);
/// Is the current token a specified keyword?
fn is_keyword(&self, kw: &str) -> bool;
}
/// `TokenCursor` abstracts the cursor of `TokenSource` operates one.
#[derive(Debug, Copy, Clone, Eq, PartialEq)]
pub struct Token {
/// What is the current token?
pub kind: SyntaxKind,
/// Is the current token joined to the next one (`> >` vs `>>`).
pub is_jointed_to_next: bool,
}
```
Note that the refactoring based on this new trait will be separated to incoming PRs
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1138: Add L_DOLLAR and R_DOLLAR r=matklad a=edwin0cheng
As discussion in issue https://github.com/rust-analyzer/rust-analyzer/issues/1132 and PR #1125 , this PR add 2 `Syntax::Kind` : `L_DOLLAR` and `R_DOLLAR` for representing `Delimiter::None` in mbe and proc_marco. By design, It should not affect the final syntax tree, and will be discard in `TreeSink`.
My original idea is handling these 2 tokens case by case, but i found that they will appear in every place in the parser (imagine `tt` matcher). So this PR only handle it in `Parser::do_bump` and `Parser::start`, although It will not fix the `expr` matcher executing order problem in original idea.
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1105: [WIP] Implement ra_mbe meta variables support r=matklad a=edwin0cheng
This PR implements the following meta variable support in `ra_mba` crate (issue #720):
- [x] `path`
- [ ] `expr`
- [ ] `ty`
- [ ] `pat`
- [ ] `stmt`
- [ ] `block`
- [ ] `meta`
- [ ] `item`
*Implementation Details*
In the macro expanding lhs phase, if we see a meta variable type, we try to create a `tt:TokenTree` from the remaining input. And then we use a special set of `ra_parser` to parse it to `SyntaxNode`.
Co-authored-by: Edwin Cheng <edwin0cheng@gmail.com>
1082: Async block in argument position r=matklad a=andreytkachenko
Fixes case when async block appears in argument position
Co-authored-by: Andrey Tkachenko <andreytkachenko64@gmail.com>
1081: Async closure syntax r=matklad a=robojumper
Fixes#1080.
Also fixes an error introduced by #1072 where something like `async move "foo"` in expression position would trigger the assertion in `block_expr`.
Co-authored-by: robojumper <robojumper@gmail.com>
Now bounds inside a path are parsed as DYN_TRAIT_TYPE, previously they would be
parsed as `PATH_TYPE` followed by `TYPE_BOUND_LIST`.
Basically this means `Box<T + 'f>` is now parsed almost the same as
`Box<dyn T + 'f>` with the exception of not having the `dyn` keyword.
These are now used when parsing type bounds. In addition parsing paths inside a
bound now does not recursively parse paths, rather they are treated as separate
bounds, separated by +.
991: Use Marker argument for item parsers r=matklad a=pcpthm
Before doing this for expressions, I found that the pattern (Marker argument) should be applied to the item parsers because visiblity and modifiers are parsed in a separate function.
Fixed some parser bugs:
- Fix pub_expr: `pub 42;` was allowed.
- Fix incorrect parsing of crate::path: incorrectly parsed as `crate` as a visibility.
Co-authored-by: pcpthm <pcpthm@gmail.com>
987: Refactor maybe_item to use Marker argument r=pcpthm a=pcpthm
As suggested at <https://github.com/rust-analyzer/rust-analyzer/pull/980#issuecomment-473659745>.
For expression paring functions, changing signature
- from `fn(&mut Parser) -> Option<CompletedMarker>` to `fn(&mut Parser, Marker) -> Result<CompletedMarker, Marker>`
- from `fn(&mut Parser) -> CompletedMarker` to `fn(&mut Parser, Marker) -> CompletedMarker`
is my plan.
Co-authored-by: pcpthm <pcpthm@gmail.com>
983: support remainder assignment operator r=matklad a=JeanMertz
`%=` was returning errors for me, turns out it wasn't added as a valid assignment operation.
I'm not sure what the best location would be to add a test for this. Please let me know and I'll add one.
Co-authored-by: Jean Mertz <jean@mertz.fm>
