They are only used by rustc_lexer, and are not needed elsewhere.
So we move the relevant definitions into rustc_lexer (while the actual
unicode data comes from the unicode-xid crate) and make the rest of
the compiler use it.
add rustc_private as a proper language feature gate
At the moment, `rustc_private` as a (library) feature exists by
accident: `char::is_xid_start`, `char::is_xid_continue` methods in
libcore define it.
cc https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/How.20to.20declare.20new.20langauge.20feature.3F
I don't know if this is at all reasonable, but at least tests seem to pass locally. That probably means that we can remove/rename to something more resonable the feature in libcore in the next release?
The idea here is to make a reusable library out of the existing
rust-lexer, by separating out pure lexing and rustc-specific concerns,
like spans, error reporting an interning.
So, rustc_lexer operates directly on `&str`, produces simple tokens
which are a pair of type-tag and a bit of original text, and does not
report errors, instead storing them as flags on the token.
