Commit Graph

10403 Commits

Author SHA1 Message Date
darksv
100968b689 Support for unions 2018-09-14 22:51:12 +02:00
darksv
bc94bf95ce correctly handle IDENTs when changed to contextual keywords 2018-09-14 19:26:48 +02:00
darksv
c300135322 create leaf directly without calling the parser 2018-09-14 19:23:10 +02:00
darksv
4356240fa4 Incremental reparsing for single tokens (WHITESPACE, COMMENT, DOC_COMMENT, IDENT, STRING, RAW_STRING) 2018-09-13 23:25:05 +02:00
Aleksey Kladov
b6f8037a6f don't get stuck in slice patterns 2018-09-12 11:26:52 +03:00
Aleksey Kladov
ccc75675b6 correctly setup path-map for fs-changes 2018-09-12 11:19:19 +03:00
bors[bot]
e240360ee2 Merge #68
68: Implement incremental reparsing for remaining braced blocks r=matklad a=darksv

Fixes #66

Co-authored-by: darksv <darek969-12@o2.pl>
2018-09-11 07:32:36 +00:00
darksv
d0cfeb4f16 Do not reparse token tree when it is not delimited by braces 2018-09-10 23:21:16 +02:00
darksv
64d07c1bd4 Implement reparsing for remaining blocks 2018-09-10 20:14:09 +02:00
Aleksey Kladov
505895a25f store file rsovler 2018-09-10 12:57:40 +03:00
bors[bot]
4f64709666 Merge #65
65: simplify r=matklad a=matklad



Co-authored-by: Aleksey Kladov <aleksey.kladov@gmail.com>
2018-09-08 16:24:24 +00:00
Aleksey Kladov
bae5938682 note about symbol search 2018-09-08 19:23:52 +03:00
Aleksey Kladov
f19a82beac simplify 2018-09-08 19:16:11 +03:00
bors[bot]
8c3720d4b8 Merge #64
64: Add fuzz test-case with a fix r=matklad a=matklad



Co-authored-by: Aleksey Kladov <aleksey.kladov@gmail.com>
2018-09-08 16:11:31 +00:00
Aleksey Kladov
a5c333c3ed Fix yet another parser infinite loop
This commit is an example of fixing a common parser error: infinite
loop due to error recovery.

This error typically happens when we parse a list of items and fail to
parse a specific item at the current position.

One choices is to skip a token and try to parse a list item at the
next position. This is a good, but not universal, default. When
parsing a list of arguments in a function call, you, for example,
don't want to skip over `fn`, because it's most likely that it is a
function declaration, and not a mistyped arg:

```
fn foo() {
    quux(1, 2

fn bar() {
}
```

Another choice is to bail out of the loop immediately, but it isn't
perfect either: sometimes skipping over garbage helps:

```
quux(1, foo:, 92) // should skip over `:`, b/c that's part of `foo::bar`
```

In general, parser tries to balance these two cases, though we don't
have a definitive strategy yet.

However, if the parser accidentally neither skips over a token, nor
breaks out of the loop, then it becomes stuck in the loop infinitely
(there's an internal counter to self-check this situation and panic
though), and that's exactly what is demonstrated by the test.

To fix such situation, first of all, add the test case to tests/data/parser/{err,fuzz-failures}.

Then, run

```
RUST_BACKTRACE=short cargo test --package libsyntax2
````

to verify that parser indeed panics, and to get an idea what grammar
production is the culprit (look for `_list` functions!).

In this case, I see

```
  10: libsyntax2::grammar::expressions::atom::match_arm_list
             at crates/libsyntax2/src/grammar/expressions/atom.rs:309
```

and that's look like it might be a culprit. I verify it by adding
`eprintln!("loopy {:?}", p.current());` and indeed I see that this is
printed repeatedly.

Diagnosing this a bit shows that the problem is that
`pattern::pattern` function does not consume anything if the next
token is `let`. That is a good default to make cases like

```
let
let foo = 92;
```

where the user hasn't typed the pattern yet, to parse in a reasonable
they correctly.

For match arms, pretty much the single thing we expect is a pattern,
so, for a fix, I introduce a special variant of pattern that does not
do recovery.
2018-09-08 19:10:40 +03:00
Aleksey Kladov
3ab9f4ad7f Add fuzz failures dir 2018-09-08 18:42:59 +03:00
Aleksey Kladov
ba4a697d8c move fuzz-invariants to the library 2018-09-08 18:34:41 +03:00
bors[bot]
a60b9ad963 Merge #63
63: Add trivial fuzzer for parser r=matklad a=killercup

As described in #61, fuzz testing some parts of this would be ~~fun~~
helpful. So, I started with the most trivial fuzzer I could think of:
Put random stuff into File::parse and see what happens.

To speed things up, I also did

    cp src/**/*.rs fuzz/corpus/parser/

in the `crates/libsyntax2/` directory (running the fuzzer once will
generate the necessary directories).

Co-authored-by: Pascal Hertleif <killercup@gmail.com>
2018-09-08 15:20:46 +00:00
Pascal Hertleif
a37cd5ad43 Add trivial fuzzer for parser
As described in #61, fuzz testing some parts of this would be ~~fun~~
helpful. So, I started with the most trivial fuzzer I could think of:
Put random stuff into File::parse and see what happens.

To speed things up, I also did

    cp src/**/*.rs fuzz/corpus/parser/

in the `crates/libsyntax2/` directory (running the fuzzer once will
generate the necessary directories).
2018-09-08 16:55:53 +02:00
Aleksey Kladov
df05c5c3e2 Don't overflow when limiting symbol search 2018-09-08 15:39:28 +03:00
Aleksey Kladov
7daaddb2ac Some abstraction around workers 2018-09-08 13:15:01 +03:00
Aleksey Kladov
326ffcefe0 Deal with deadlocks in a more principaled way 2018-09-08 12:36:02 +03:00
Aleksey Kladov
d9ccebd913 fix deadlock 2018-09-08 12:08:46 +03:00
Aleksey Kladov
f48b9d9be7 Fix block structure in enums 2018-09-08 10:55:09 +03:00
Aleksey Kladov
749907d330 simplify 2018-09-08 10:38:53 +03:00
Aleksey Kladov
febbc9acdd Don't get stuck in tuple exprs 2018-09-08 10:35:05 +03:00
Aleksey Kladov
a0a347eac9 Don't get stuck in macros 2018-09-08 10:28:53 +03:00
Aleksey Kladov
bd3a26493f fix stuck parser 2018-09-08 10:13:32 +03:00
Aleksey Kladov
44334f6f56 fix labled expressions 2018-09-08 09:18:42 +03:00
Aleksey Kladov
127814d9a7 nested mod completion 2018-09-08 01:35:20 +03:00
Aleksey Kladov
ff1c82216c Remove dyn dispatch 2018-09-08 01:16:07 +03:00
Aleksey Kladov
fcfda94664 Separete API from IMPL
Looks like there's a rule of thumb: don't call API functions from an
implementation! In this case, following this rule of thumb saves us an
Arc-bump!
2018-09-07 22:05:05 +03:00
bors[bot]
ba7b3c2108 Merge #59
59: Moved TokenSet into it's own file. r=matklad a=Plasticcaz

As discussed in Issue #11, the only thing left in that issue that hasn't been fixed appears to be that TokenSet is not in it's own file. This pull request pulls TokenSet, it's macros and it's test into it's own file.

Co-authored-by: Zac Winter <plasticcaz@gmail.com>
2018-09-06 14:05:37 +00:00
Zac Winter
518cc87496 Moved TokenSet into it's own file. 2018-09-06 21:57:04 +08:00
Aleksey Kladov
751562d2f7 better introduce 2018-09-06 01:19:24 +03:00
Aleksey Kladov
bb64edf8ba introduce variable 2018-09-06 00:59:07 +03:00
Aleksey Kladov
47e8b80e9b use correct workdir for the server 2018-09-05 21:38:43 +03:00
bors[bot]
529ebd5840 Merge #58
58: even less hacks r=matklad a=matklad



Co-authored-by: Aleksey Kladov <aleksey.kladov@gmail.com>
2018-09-05 17:23:25 +00:00
Aleksey Kladov
669eabe892 even less hacks 2018-09-05 20:22:52 +03:00
bors[bot]
8f30179f82 Merge #57
57: less hacky paths r=matklad a=matklad



Co-authored-by: Aleksey Kladov <aleksey.kladov@gmail.com>
2018-09-05 15:28:21 +00:00
Aleksey Kladov
d0e22d7578 less hacky paths 2018-09-05 18:27:44 +03:00
bors[bot]
ad451686a8 Merge #56
56: Unify lookahead naming between parser and lexer. r=matklad a=zachlute

Resolves Issue #26.

I wanted to play around with libsyntax2, and fixing a random issue seemed like a good way to mess around in the code.

This PR mostly does what's suggested in that issue. I elected to go with `at` and `at_str` instead of trying to do any fancy overloading shenanigans, because...uh, well, frankly I don't really know how to do any fancy overloading shenanigans. The only really questionable bit is `nth_is_p`, which could also have potentially been named `nth_at_p`, but `is` seemed more apropos.

I also added simple tests for `Ptr` so I could be less terrified I broke something. 

Comments and criticisms very welcome. I'm still pretty new to Rust.

Co-authored-by: Zach Lute <zach.lute@gmail.com>
2018-09-05 15:07:17 +00:00
Aleksey Kladov
649f7faf7d fix tests on windows 2018-09-05 15:03:27 +01:00
Zach Lute
d21fead150 Added tests for Ptr. 2018-09-04 23:26:11 -07:00
Zach Lute
af0ae9ee04 Updated Ptr methods to better match Parser method names. 2018-09-04 22:56:16 -07:00
Aleksey Kladov
f87771092c switch to rayon threadpool 2018-09-04 20:43:37 +03:00
Aleksey Kladov
8b0210d233 simplify 2018-09-04 19:00:01 +03:00
Aleksey Kladov
8ed06d766f better extend selection 2018-09-04 12:48:39 +03:00
Aleksey Kladov
e44a6bcc82 for types in bounds 2018-09-04 12:25:23 +03:00
Aleksey Kladov
3a017aaa52 dont change readonly files 2018-09-04 11:40:45 +03:00