2021-12-25 12:59:02 -06:00
|
|
|
//! See [`Output`]
|
|
|
|
|
internal: replace TreeSink with a data structure
The general theme of this is to make parser a better independent
library.
The specific thing we do here is replacing callback based TreeSink with
a data structure. That is, rather than calling user-provided tree
construction methods, the parser now spits out a very bare-bones tree,
effectively a log of a DFS traversal.
This makes the parser usable without any *specifc* tree sink, and allows
us to, eg, move tests into this crate.
Now, it's also true that this is a distinction without a difference, as
the old and the new interface are equivalent in expressiveness. Still,
this new thing seems somewhat simpler. But yeah, I admit I don't have a
suuper strong motivation here, just a hunch that this is better.
2021-12-19 08:36:23 -06:00
|
|
|
use crate::SyntaxKind;
|
|
|
|
|
2021-12-25 12:59:02 -06:00
|
|
|
/// Output of the parser -- a DFS traversal of a concrete syntax tree.
|
|
|
|
///
|
|
|
|
/// Use the [`Output::iter`] method to iterate over traversal steps and consume
|
|
|
|
/// a syntax tree.
|
|
|
|
///
|
|
|
|
/// In a sense, this is just a sequence of [`SyntaxKind`]-colored parenthesis
|
|
|
|
/// interspersed into the original [`crate::Input`]. The output is fundamentally
|
|
|
|
/// coordinated with the input and `n_input_tokens` refers to the number of
|
|
|
|
/// times [`crate::Input::push`] was called.
|
internal: replace TreeSink with a data structure
The general theme of this is to make parser a better independent
library.
The specific thing we do here is replacing callback based TreeSink with
a data structure. That is, rather than calling user-provided tree
construction methods, the parser now spits out a very bare-bones tree,
effectively a log of a DFS traversal.
This makes the parser usable without any *specifc* tree sink, and allows
us to, eg, move tests into this crate.
Now, it's also true that this is a distinction without a difference, as
the old and the new interface are equivalent in expressiveness. Still,
this new thing seems somewhat simpler. But yeah, I admit I don't have a
suuper strong motivation here, just a hunch that this is better.
2021-12-19 08:36:23 -06:00
|
|
|
#[derive(Default)]
|
2021-12-25 12:59:02 -06:00
|
|
|
pub struct Output {
|
internal: replace TreeSink with a data structure
The general theme of this is to make parser a better independent
library.
The specific thing we do here is replacing callback based TreeSink with
a data structure. That is, rather than calling user-provided tree
construction methods, the parser now spits out a very bare-bones tree,
effectively a log of a DFS traversal.
This makes the parser usable without any *specifc* tree sink, and allows
us to, eg, move tests into this crate.
Now, it's also true that this is a distinction without a difference, as
the old and the new interface are equivalent in expressiveness. Still,
this new thing seems somewhat simpler. But yeah, I admit I don't have a
suuper strong motivation here, just a hunch that this is better.
2021-12-19 08:36:23 -06:00
|
|
|
/// 32-bit encoding of events. If LSB is zero, then that's an index into the
|
|
|
|
/// error vector. Otherwise, it's one of the thee other variants, with data encoded as
|
|
|
|
///
|
2021-12-25 12:59:02 -06:00
|
|
|
/// |16 bit kind|8 bit n_input_tokens|4 bit tag|4 bit leftover|
|
internal: replace TreeSink with a data structure
The general theme of this is to make parser a better independent
library.
The specific thing we do here is replacing callback based TreeSink with
a data structure. That is, rather than calling user-provided tree
construction methods, the parser now spits out a very bare-bones tree,
effectively a log of a DFS traversal.
This makes the parser usable without any *specifc* tree sink, and allows
us to, eg, move tests into this crate.
Now, it's also true that this is a distinction without a difference, as
the old and the new interface are equivalent in expressiveness. Still,
this new thing seems somewhat simpler. But yeah, I admit I don't have a
suuper strong motivation here, just a hunch that this is better.
2021-12-19 08:36:23 -06:00
|
|
|
///
|
|
|
|
event: Vec<u32>,
|
|
|
|
error: Vec<String>,
|
|
|
|
}
|
|
|
|
|
2021-12-29 09:23:34 -06:00
|
|
|
#[derive(Debug)]
|
2021-12-25 12:59:02 -06:00
|
|
|
pub enum Step<'a> {
|
|
|
|
Token { kind: SyntaxKind, n_input_tokens: u8 },
|
2023-02-07 11:08:05 -06:00
|
|
|
FloatSplit { ends_in_dot: bool },
|
2021-12-25 12:59:02 -06:00
|
|
|
Enter { kind: SyntaxKind },
|
|
|
|
Exit,
|
internal: replace TreeSink with a data structure
The general theme of this is to make parser a better independent
library.
The specific thing we do here is replacing callback based TreeSink with
a data structure. That is, rather than calling user-provided tree
construction methods, the parser now spits out a very bare-bones tree,
effectively a log of a DFS traversal.
This makes the parser usable without any *specifc* tree sink, and allows
us to, eg, move tests into this crate.
Now, it's also true that this is a distinction without a difference, as
the old and the new interface are equivalent in expressiveness. Still,
this new thing seems somewhat simpler. But yeah, I admit I don't have a
suuper strong motivation here, just a hunch that this is better.
2021-12-19 08:36:23 -06:00
|
|
|
Error { msg: &'a str },
|
|
|
|
}
|
|
|
|
|
2021-12-25 12:59:02 -06:00
|
|
|
impl Output {
|
2023-02-03 04:47:33 -06:00
|
|
|
const EVENT_MASK: u32 = 0b1;
|
|
|
|
const TAG_MASK: u32 = 0x0000_00F0;
|
|
|
|
const N_INPUT_TOKEN_MASK: u32 = 0x0000_FF00;
|
|
|
|
const KIND_MASK: u32 = 0xFFFF_0000;
|
|
|
|
|
|
|
|
const ERROR_SHIFT: u32 = Self::EVENT_MASK.trailing_ones();
|
|
|
|
const TAG_SHIFT: u32 = Self::TAG_MASK.trailing_zeros();
|
|
|
|
const N_INPUT_TOKEN_SHIFT: u32 = Self::N_INPUT_TOKEN_MASK.trailing_zeros();
|
|
|
|
const KIND_SHIFT: u32 = Self::KIND_MASK.trailing_zeros();
|
|
|
|
|
|
|
|
const TOKEN_EVENT: u8 = 0;
|
|
|
|
const ENTER_EVENT: u8 = 1;
|
|
|
|
const EXIT_EVENT: u8 = 2;
|
2023-02-03 10:18:48 -06:00
|
|
|
const SPLIT_EVENT: u8 = 3;
|
2023-02-03 04:47:33 -06:00
|
|
|
|
2021-12-25 12:59:02 -06:00
|
|
|
pub fn iter(&self) -> impl Iterator<Item = Step<'_>> {
|
internal: replace TreeSink with a data structure
The general theme of this is to make parser a better independent
library.
The specific thing we do here is replacing callback based TreeSink with
a data structure. That is, rather than calling user-provided tree
construction methods, the parser now spits out a very bare-bones tree,
effectively a log of a DFS traversal.
This makes the parser usable without any *specifc* tree sink, and allows
us to, eg, move tests into this crate.
Now, it's also true that this is a distinction without a difference, as
the old and the new interface are equivalent in expressiveness. Still,
this new thing seems somewhat simpler. But yeah, I admit I don't have a
suuper strong motivation here, just a hunch that this is better.
2021-12-19 08:36:23 -06:00
|
|
|
self.event.iter().map(|&event| {
|
2023-02-03 04:47:33 -06:00
|
|
|
if event & Self::EVENT_MASK == 0 {
|
|
|
|
return Step::Error {
|
|
|
|
msg: self.error[(event as usize) >> Self::ERROR_SHIFT].as_str(),
|
|
|
|
};
|
internal: replace TreeSink with a data structure
The general theme of this is to make parser a better independent
library.
The specific thing we do here is replacing callback based TreeSink with
a data structure. That is, rather than calling user-provided tree
construction methods, the parser now spits out a very bare-bones tree,
effectively a log of a DFS traversal.
This makes the parser usable without any *specifc* tree sink, and allows
us to, eg, move tests into this crate.
Now, it's also true that this is a distinction without a difference, as
the old and the new interface are equivalent in expressiveness. Still,
this new thing seems somewhat simpler. But yeah, I admit I don't have a
suuper strong motivation here, just a hunch that this is better.
2021-12-19 08:36:23 -06:00
|
|
|
}
|
2023-02-03 04:47:33 -06:00
|
|
|
let tag = ((event & Self::TAG_MASK) >> Self::TAG_SHIFT) as u8;
|
internal: replace TreeSink with a data structure
The general theme of this is to make parser a better independent
library.
The specific thing we do here is replacing callback based TreeSink with
a data structure. That is, rather than calling user-provided tree
construction methods, the parser now spits out a very bare-bones tree,
effectively a log of a DFS traversal.
This makes the parser usable without any *specifc* tree sink, and allows
us to, eg, move tests into this crate.
Now, it's also true that this is a distinction without a difference, as
the old and the new interface are equivalent in expressiveness. Still,
this new thing seems somewhat simpler. But yeah, I admit I don't have a
suuper strong motivation here, just a hunch that this is better.
2021-12-19 08:36:23 -06:00
|
|
|
match tag {
|
2023-02-03 04:47:33 -06:00
|
|
|
Self::TOKEN_EVENT => {
|
|
|
|
let kind: SyntaxKind =
|
|
|
|
(((event & Self::KIND_MASK) >> Self::KIND_SHIFT) as u16).into();
|
|
|
|
let n_input_tokens =
|
|
|
|
((event & Self::N_INPUT_TOKEN_MASK) >> Self::N_INPUT_TOKEN_SHIFT) as u8;
|
2021-12-25 12:59:02 -06:00
|
|
|
Step::Token { kind, n_input_tokens }
|
internal: replace TreeSink with a data structure
The general theme of this is to make parser a better independent
library.
The specific thing we do here is replacing callback based TreeSink with
a data structure. That is, rather than calling user-provided tree
construction methods, the parser now spits out a very bare-bones tree,
effectively a log of a DFS traversal.
This makes the parser usable without any *specifc* tree sink, and allows
us to, eg, move tests into this crate.
Now, it's also true that this is a distinction without a difference, as
the old and the new interface are equivalent in expressiveness. Still,
this new thing seems somewhat simpler. But yeah, I admit I don't have a
suuper strong motivation here, just a hunch that this is better.
2021-12-19 08:36:23 -06:00
|
|
|
}
|
2023-02-03 04:47:33 -06:00
|
|
|
Self::ENTER_EVENT => {
|
|
|
|
let kind: SyntaxKind =
|
|
|
|
(((event & Self::KIND_MASK) >> Self::KIND_SHIFT) as u16).into();
|
2021-12-25 12:59:02 -06:00
|
|
|
Step::Enter { kind }
|
internal: replace TreeSink with a data structure
The general theme of this is to make parser a better independent
library.
The specific thing we do here is replacing callback based TreeSink with
a data structure. That is, rather than calling user-provided tree
construction methods, the parser now spits out a very bare-bones tree,
effectively a log of a DFS traversal.
This makes the parser usable without any *specifc* tree sink, and allows
us to, eg, move tests into this crate.
Now, it's also true that this is a distinction without a difference, as
the old and the new interface are equivalent in expressiveness. Still,
this new thing seems somewhat simpler. But yeah, I admit I don't have a
suuper strong motivation here, just a hunch that this is better.
2021-12-19 08:36:23 -06:00
|
|
|
}
|
2023-02-03 04:47:33 -06:00
|
|
|
Self::EXIT_EVENT => Step::Exit,
|
2023-02-03 10:18:48 -06:00
|
|
|
Self::SPLIT_EVENT => {
|
2023-02-07 11:08:05 -06:00
|
|
|
Step::FloatSplit { ends_in_dot: event & Self::N_INPUT_TOKEN_MASK != 0 }
|
2023-02-03 10:18:48 -06:00
|
|
|
}
|
internal: replace TreeSink with a data structure
The general theme of this is to make parser a better independent
library.
The specific thing we do here is replacing callback based TreeSink with
a data structure. That is, rather than calling user-provided tree
construction methods, the parser now spits out a very bare-bones tree,
effectively a log of a DFS traversal.
This makes the parser usable without any *specifc* tree sink, and allows
us to, eg, move tests into this crate.
Now, it's also true that this is a distinction without a difference, as
the old and the new interface are equivalent in expressiveness. Still,
this new thing seems somewhat simpler. But yeah, I admit I don't have a
suuper strong motivation here, just a hunch that this is better.
2021-12-19 08:36:23 -06:00
|
|
|
_ => unreachable!(),
|
|
|
|
}
|
|
|
|
})
|
|
|
|
}
|
|
|
|
|
|
|
|
pub(crate) fn token(&mut self, kind: SyntaxKind, n_tokens: u8) {
|
2023-02-03 04:47:33 -06:00
|
|
|
let e = ((kind as u16 as u32) << Self::KIND_SHIFT)
|
|
|
|
| ((n_tokens as u32) << Self::N_INPUT_TOKEN_SHIFT)
|
|
|
|
| Self::EVENT_MASK;
|
internal: replace TreeSink with a data structure
The general theme of this is to make parser a better independent
library.
The specific thing we do here is replacing callback based TreeSink with
a data structure. That is, rather than calling user-provided tree
construction methods, the parser now spits out a very bare-bones tree,
effectively a log of a DFS traversal.
This makes the parser usable without any *specifc* tree sink, and allows
us to, eg, move tests into this crate.
Now, it's also true that this is a distinction without a difference, as
the old and the new interface are equivalent in expressiveness. Still,
this new thing seems somewhat simpler. But yeah, I admit I don't have a
suuper strong motivation here, just a hunch that this is better.
2021-12-19 08:36:23 -06:00
|
|
|
self.event.push(e)
|
|
|
|
}
|
|
|
|
|
2023-02-07 11:08:05 -06:00
|
|
|
pub(crate) fn float_split_hack(&mut self, ends_in_dot: bool) {
|
2023-02-03 10:18:48 -06:00
|
|
|
let e = (Self::SPLIT_EVENT as u32) << Self::TAG_SHIFT
|
2023-02-07 11:08:05 -06:00
|
|
|
| ((ends_in_dot as u32) << Self::N_INPUT_TOKEN_SHIFT)
|
2023-02-03 10:18:48 -06:00
|
|
|
| Self::EVENT_MASK;
|
|
|
|
self.event.push(e);
|
|
|
|
}
|
|
|
|
|
internal: replace TreeSink with a data structure
The general theme of this is to make parser a better independent
library.
The specific thing we do here is replacing callback based TreeSink with
a data structure. That is, rather than calling user-provided tree
construction methods, the parser now spits out a very bare-bones tree,
effectively a log of a DFS traversal.
This makes the parser usable without any *specifc* tree sink, and allows
us to, eg, move tests into this crate.
Now, it's also true that this is a distinction without a difference, as
the old and the new interface are equivalent in expressiveness. Still,
this new thing seems somewhat simpler. But yeah, I admit I don't have a
suuper strong motivation here, just a hunch that this is better.
2021-12-19 08:36:23 -06:00
|
|
|
pub(crate) fn enter_node(&mut self, kind: SyntaxKind) {
|
2023-02-03 04:47:33 -06:00
|
|
|
let e = ((kind as u16 as u32) << Self::KIND_SHIFT)
|
|
|
|
| ((Self::ENTER_EVENT as u32) << Self::TAG_SHIFT)
|
|
|
|
| Self::EVENT_MASK;
|
internal: replace TreeSink with a data structure
The general theme of this is to make parser a better independent
library.
The specific thing we do here is replacing callback based TreeSink with
a data structure. That is, rather than calling user-provided tree
construction methods, the parser now spits out a very bare-bones tree,
effectively a log of a DFS traversal.
This makes the parser usable without any *specifc* tree sink, and allows
us to, eg, move tests into this crate.
Now, it's also true that this is a distinction without a difference, as
the old and the new interface are equivalent in expressiveness. Still,
this new thing seems somewhat simpler. But yeah, I admit I don't have a
suuper strong motivation here, just a hunch that this is better.
2021-12-19 08:36:23 -06:00
|
|
|
self.event.push(e)
|
|
|
|
}
|
|
|
|
|
|
|
|
pub(crate) fn leave_node(&mut self) {
|
2023-02-03 04:47:33 -06:00
|
|
|
let e = (Self::EXIT_EVENT as u32) << Self::TAG_SHIFT | Self::EVENT_MASK;
|
internal: replace TreeSink with a data structure
The general theme of this is to make parser a better independent
library.
The specific thing we do here is replacing callback based TreeSink with
a data structure. That is, rather than calling user-provided tree
construction methods, the parser now spits out a very bare-bones tree,
effectively a log of a DFS traversal.
This makes the parser usable without any *specifc* tree sink, and allows
us to, eg, move tests into this crate.
Now, it's also true that this is a distinction without a difference, as
the old and the new interface are equivalent in expressiveness. Still,
this new thing seems somewhat simpler. But yeah, I admit I don't have a
suuper strong motivation here, just a hunch that this is better.
2021-12-19 08:36:23 -06:00
|
|
|
self.event.push(e)
|
|
|
|
}
|
|
|
|
|
|
|
|
pub(crate) fn error(&mut self, error: String) {
|
|
|
|
let idx = self.error.len();
|
|
|
|
self.error.push(error);
|
2023-02-03 04:47:33 -06:00
|
|
|
let e = (idx as u32) << Self::ERROR_SHIFT;
|
internal: replace TreeSink with a data structure
The general theme of this is to make parser a better independent
library.
The specific thing we do here is replacing callback based TreeSink with
a data structure. That is, rather than calling user-provided tree
construction methods, the parser now spits out a very bare-bones tree,
effectively a log of a DFS traversal.
This makes the parser usable without any *specifc* tree sink, and allows
us to, eg, move tests into this crate.
Now, it's also true that this is a distinction without a difference, as
the old and the new interface are equivalent in expressiveness. Still,
this new thing seems somewhat simpler. But yeah, I admit I don't have a
suuper strong motivation here, just a hunch that this is better.
2021-12-19 08:36:23 -06:00
|
|
|
self.event.push(e);
|
|
|
|
}
|
|
|
|
}
|