e70c8584d5
This reverts commit 9c7d5ae57c27ebfc019c2c23283bb905d8c3b74f. This was wrong... the `continue` was to ignore the latter half of the tokens file. Another mechanism will have to be used to keep the model grammar's tokens in sync with the actual grammar's tokens :-/
Reference grammar.
Uses antlr4 and a custom Rust tool to compare
ASTs/token streams generated. You can use the check-lexer make target to
run all of the available tests.
To use manually:
antlr4 RustLexer.g4
javac *.java
rustc -O verify.rs
for file in ../*/**.rs; do
echo $file;
grun RustLexer tokens -tokens < "$file" | ./verify "$file" RustLexer.tokens || break
done
Note That the ../*/**.rs glob will match every *.rs file in the above
directory and all of its recursive children. This is a zsh extension.