macros: Make metavariables hygienic
This PR makes metavariables hygienic. For example, consider:
```rust
macro_rules! foo {
($x:tt) => { // Suppose that this token tree argument is always a metavariable.
macro_rules! bar { ($x:expr, $y:expr) => { ($x, $y) } }
}
}
fn main() {
foo!($z); // This currently compiles.
foo!($y); // This is an error today but compiles after this PR.
}
```
Today, the `macro_rules! bar { ... }` definition is only valid when the metavariable passed to `foo` is not `$y` (since it unhygienically conflicts with the `$y` in the definition of `bar`) or `$x` (c.f. #35450).
After this PR, the definition of `bar` is always valid (and `bar!(a, b)` always expands to `(a, b)` as expected).
This can break code that was allowed in #34925 (landed two weeks ago). For example,
```rust
macro_rules! outer {
($t:tt) => {
macro_rules! inner { ($i:item) => { $t } }
}
}
outer!($i); // This `$i` should not interact with the `$i` in the definition of `inner!`.
inner!(fn main() {}); // After this PR, this is an error ("unknown macro variable `i`").
```
Due to the severe limitations on nested `macro_rules!` before #34925, this is not a breaking change for stable/beta.
Fixes#35450.
r? @nrc
Better attribute and metaitem encapsulation throughout the compiler
This PR refactors most (hopefully all?) of the `MetaItem` interactions outside of `libsyntax` (and a few inside) to interact with MetaItems through the provided traits instead of directly creating / destruct / matching against them. This is a necessary first step to eventually converting `MetaItem`s to internally use `TokenStream` representations (which will make `MetaItem` interactions much nicer for macro writers once the new macro system is in place).
r? @nrc
Simplify the macro hygiene algorithm
This PR removes renaming from the hygiene algorithm and treats differently marked identifiers as unequal.
This change makes the scope of identifiers in `macro_rules!` items empty. That is, identifiers in `macro_rules!` definitions do not inherit any semantics from the `macro_rules!`'s scope.
Since `macro_rules!` macros are items, the scope of their identifiers "should" be the same as that of other items; in particular, the scope should contain only items. Since all items are unhygienic today, this would mean the scope should be empty.
However, the scope of an identifier in a `macro_rules!` statement today is the scope that the identifier would have if it replaced the `macro_rules!` (excluding anything unhygienic, i.e. locals only).
To continue to support this, this PR tracks the scope of each `macro_rules!` and uses it in `resolve` to ensure that an identifier expanded from a `macro_rules!` gets a chance to resolve to the locals in the `macro_rules!`'s scope.
This PR is a pure refactoring. After this PR,
- `syntax::ext::expand` is much simpler.
- We can expand macros in any order without causing problems for hygiene (needed for macro modularization).
- We can deprecate or remove today's `macro_rules!` scope easily.
- Expansion performance improves by 25%, post-expansion memory usage decreases by ~5%.
- Expanding a block is no longer quadratic in the number of `let` statements (fixes#10607).
r? @nrc
Fix expansion performance regression
**syntax-[breaking-change] cc #31645**
This fixes#34630 by reverting commit 5bf7970 of PR #33943, which landed in #34424.
By removing the `Rc<_>` wrapping around `Delimited` and `SequenceRepetition` in `TokenTree`, 5bf7970 made cloning `TokenTree`s more expensive. While this had no measurable performance impact on the compiler's crates, it caused an order of magnitude performance regression on some macro-heavy code in the wild. I believe this is due to clones of `TokenTree`s in `macro_parser.rs` and/or `macro_rules.rs`.
r? @nrc
This is a spiritual succesor to #34268/8531d581, in which we replaced a
number of matches of None to the unit value with `if let` conditionals
where it was judged that this made for clearer/simpler code (as would be
recommended by Manishearth/rust-clippy's `single_match` lint). The same
rationale applies to matches of None to the empty block.
Forbid type parameters and global paths in macro invocations
Fixes#28558.
This is a [breaking-change]. For example, the following would break:
```rust
macro_rules! m { () => { () } }
fn main() {
m::<T>!(); // Type parameters are no longer allowed in macro invocations
::m!(); // Global paths are no longer allowed in macro invocations
}
```
Any breakage can be fixed by removing the type parameters or the leading `::` (respectively).
r? @eddyb
Batch up libsyntax breaking changes
Batch of the following syntax-[breaking-change] changes:
- #34213: Add a variant `Macro` to `TraitItemKind`
- #34368: Merge the variant `QPath` of `PatKind` into the variant `PatKind::Path`
- #34385: Move `syntax::ast::TokenTree` into a new module `syntax::tokenstream`
- #33943:
- Remove the type parameter from `visit::Visitor`
- Remove `attr::WithAttrs` -- use `attr::HasAttrs` instead.
- Change `fold_tt`/`fold_tts` to take token trees by value and avoid wrapping token trees in `Rc`.
- Remove the field `ctxt` of `ast::Mac_`
- Remove inherent method `attrs()` of types -- use the method `attrs` of `HasAttrs` instead.
- #34316:
- Remove `ast::Decl`/`ast::DeclKind` and add variants `Local` and `Item` to `StmtKind`.
- Move the node id for statements from the `StmtKind` variants to a field of `Stmt` (making `Stmt` a struct instead of an alias for `Spanned<StmtKind>`)
- Rename `ast::ExprKind::Again` to `Continue`.
- #34339: Generalize and abstract `ThinAttributes` to `ThinVec<Attribute>`
- Use `.into()` in convert between `Vec<Attribute>` and `ThinVec<Attribute>`
- Use autoderef instead of `.as_attr_slice()`
- #34436: Remove the optional expression from `ast::Block` and instead use a `StmtKind::Expr` at the end of the statement list.
- #34403: Move errors into a separate crate (unlikely to cause breakage)
To allow these braced macro invocation, this PR removes the optional expression from `ast::Block` and instead uses a `StmtKind::Expr` at the end of the statement list.
Currently, braced macro invocations in blocks can expand into statements (and items) except when they are last in a block, in which case they can only expand into expressions.
For example,
```rust
macro_rules! make_stmt {
() => { let x = 0; }
}
fn f() {
make_stmt! {} //< This is OK...
let x = 0; //< ... unless this line is commented out.
}
```
Fixes#34418.
syntax-[breaking-change] cc #31645
(Only breaking because ast::TokenTree is now tokenstream::TokenTree.)
This pull request refactors TokenTrees into their own file as src/libsyntax/tokenstream.rs, moving them out of src/libsyntax/ast.rs, in order to prepare for an accompanying TokenStream implementation (per RFC 1566).
This PR refactors the 'errors' part of libsyntax into its own crate (librustc_errors). This is the first part of a few refactorings to simplify error reporting and potentially support more output formats (like a standardized JSON output and possibly an --explain mode that can work with the user's code), though this PR stands on its own and doesn't assume further changes.
As part of separating out the errors crate, I have also refactored the code position portion of codemap into its own crate (libsyntax_pos). While it's helpful to have the common code positions in a separate crate for the new errors crate, this may also enable further simplifications in the future.
**syntax-[breaking-change]** cc #31645
New `TraitItemKind::Macro` variant
This change adds support for macro expansion inside trait items by adding the new `TraitItemKind::Macro` and associated parsing code.
Add an abs_path member to FileMap, use it when writing debug info.
Fixes#34179.
When items are inlined from extern crates, the filename in the debug info
is taken from the FileMap that's serialized in the rlib metadata.
Currently this is just FileMap.name, which is whatever path is passed to rustc.
Since libcore and libstd are built by invoking rustc with relative paths,
they wind up with relative paths in the rlib, and when linked into a binary
the debug info uses relative paths for the names, but since the compilation
directory for the final binary, tools trying to read source filenames
will wind up with bad paths. We noticed this in Firefox with source
filenames from libcore/libstd having bad paths.
This change stores an absolute path in FileMap.abs_path, and uses that
if available for writing debug info. This is not going to magically make
debuggers able to find the source, but it will at least provide sensible
paths.
When items are inlined from extern crates, the filename in the debug info
is taken from the FileMap that's serialized in the rlib metadata.
Currently this is just FileMap.name, which is whatever path is passed to rustc.
Since libcore and libstd are built by invoking rustc with relative paths,
they wind up with relative paths in the rlib, and when linked into a binary
the debug info uses relative paths for the names, but since the compilation
directory for the final binary, tools trying to read source filenames
will wind up with bad paths. We noticed this in Firefox with source
filenames from libcore/libstd having bad paths.
This change stores an absolute path in FileMap.abs_path, and uses that
if available for writing debug info. This is not going to magically make
debuggers able to find the source, but it will at least provide sensible
paths.
Revert a change in the scope of macros imported from crates to fix a regression
Fixes#34212.
The regression was caused by #34032, which changed the scope of macros imported from extern crates to match the scope of macros imported from modules.
r? @nrc
Remove the old FOLLOW checking (aka `check_matcher_old`).
It was supposed to be removed at the next release cycle but is still in the tree since like 6 months.
Potential breaking change, since some cases (such as #25658) will change from a warning to an error. But the warning stating that it will be a hard error in the next release has been there for 6 months now.
I think it's safe to break this code. ^_^
Projection cache and better warnings for #32330
This PR does three things:
- it lays the groundwork for the more precise subtyping rules discussed in #32330, but does not enable them;
- it issues warnings when the result of a leak-check or subtyping check relies on a late-bound region which will late become early-bound when #32330 is fixed;
- it introduces a cache for projection in the inference context.
I'm not 100% happy with the approach taken by the cache here, but it seems like a step in the right direction. It results in big wins on some test cases, but not as big as previous versions -- I think because it is caching the `Vec<Obligation>` (whereas before I just returned the normalized type with an empty vector). However, that change was needed to fix an ICE in @alexcrichton's future-rs module (I haven't fully tracked the cause of that ICE yet). Also, because trans/the collector use a fresh inference context for every call to `fulfill_obligation`, they don't profit nearly as much from this cache as they ought to.
Still, here are the results from the future-rs `retry.rs`:
```
06:26 <nmatsakis> time: 6.246; rss: 44MB item-bodies checking
06:26 <nmatsakis> time: 54.783; rss: 63MB translation item collection
06:26 <nmatsakis> time: 140.086; rss: 86MB translation
06:26 <nmatsakis> time: 0.361; rss: 46MB item-bodies checking
06:26 <nmatsakis> time: 5.299; rss: 63MB translation item collection
06:26 <nmatsakis> time: 12.140; rss: 86MB translation
```
~~Another example is the example from #31849. For that, I get 34s to run item-bodies without any cache. The version of the cache included here takes 2s to run item-bodies type-checking. An alternative version which doesn't track nested obligations takes 0.2s, but that version ICEs on @alexcrichton's future-rs (and may well be incorrect, I've not fully convinced myself of that). So, a definite win, but I think there's definitely room for further progress.~~
Pushed a modified version which improves performance of the case from #31849:
```
lunch-box. time rustc --stage0 ~/tmp/issue-31849.rs -Z no-trans
real 0m33.539s
user 0m32.932s
sys 0m0.570s
lunch-box. time rustc --stage2 ~/tmp/issue-31849.rs -Z no-trans
real 0m0.195s
user 0m0.154s
sys 0m0.042s
```
Some sort of cache is also needed for unblocking further work on lazy normalization, since that will lean even more heavily on the cache, and will also require cycle detection.
r? @arielb1
Reject a LHS formed of a single sequence TT during `macro_rules!` checking.
This was already rejected during expansion. Encountering malformed LHS or RHS during expansion is now considered a bug.
Follow up to #33689.
r? @pnkfelix
Note: this can break code that defines such macros but does not use them.
Perform `cfg` attribute processing during macro expansion and fix bugs
This PR refactors `cfg` attribute processing and fixes bugs. More specifically:
- It merges gated feature checking for stmt/expr attributes, `cfg_attr` processing, and `cfg` processing into a single fold.
- This allows feature gated `cfg` variables to be used in `cfg_attr` on unconfigured items. All other feature gated attributes can already be used on unconfigured items.
- It performs `cfg` attribute processing during macro expansion instead of after expansion so that macro-expanded items are configured the same as ordinary items. In particular, to match their non-expanded counterparts,
- macro-expanded unconfigured macro invocations are no longer expanded,
- macro-expanded unconfigured macro definitions are no longer usable, and
- feature gated `cfg` variables on macro-expanded macro definitions/invocations are now errors.
This is a [breaking-change]. For example, the following would break:
```rust
macro_rules! m {
() => {
#[cfg(attr)]
macro_rules! foo { () => {} }
foo!(); // This will be an error
macro_rules! bar { () => { fn f() {} } }
#[cfg(attr)] bar!(); // This will no longer be expanded ...
fn g() { f(); } // ... so that `f` will be unresolved.
#[cfg(target_thread_local)] // This will be a gated feature error
macro_rules! baz { () => {} }
}
}
m!();
```
r? @nrc
This makes the \"shadowing labels\" warning *not* print the entire loop as a span, but only the lifetime.
Also makes #31719 go away, but does not fix its root cause (the span of the expanded loop is still wonky, but not used anymore).
Make sure that macros that didn't pass LHS checking are not expanded.
This avoid duplicate errors for things like invalid fragment specifiers, or
parsing errors for ambiguous macros.
This makes the "shadowing labels" warning *not* print the entire loop
as a span, but only the lifetime.
Also makes #31719 go away, but does not fix its root cause (the span
of the expanded loop is still wonky, but not used anymore).
This commit is an implementation of [RFC 1513] which allows applications to
alter the behavior of panics at compile time. A new compiler flag, `-C panic`,
is added and accepts the values `unwind` or `panic`, with the default being
`unwind`. This model affects how code is generated for the local crate, skipping
generation of landing pads with `-C panic=abort`.
[RFC 1513]: https://github.com/rust-lang/rfcs/blob/master/text/1513-less-unwinding.md
Panic implementations are then provided by crates tagged with
`#![panic_runtime]` and lazily required by crates with
`#![needs_panic_runtime]`. The panic strategy (`-C panic` value) of the panic
runtime must match the final product, and if the panic strategy is not `abort`
then the entire DAG must have the same panic strategy.
With the `-C panic=abort` strategy, users can expect a stable method to disable
generation of landing pads, improving optimization in niche scenarios,
decreasing compile time, and decreasing output binary size. With the `-C
panic=unwind` strategy users can expect the existing ability to isolate failure
in Rust code from the outside world.
Organizationally, this commit dismantles the `sys_common::unwind` module in
favor of some bits moving part of it to `libpanic_unwind` and the rest into the
`panicking` module in libstd. The custom panic runtime support is pretty similar
to the custom allocator support with the only major difference being how the
panic runtime is injected (takes the `-C panic` flag into account).
The extra filename and line was mainly there to keep the indentation
relative to the main snippet; now that this doesn't include
filename/line-number as a prefix, it is distracted.
Feature gate clean
This PR does a bit of cleaning in the feature-gate-handling code of libsyntax. It also fixes two bugs (#32782 and #32648). Changes include:
* Change the way the existing features are declared in `feature_gate.rs`. The array of features and the `Features` struct are now defined together by a single macro. `featureck.py` has been updated accordingly. Note: there are now three different arrays for active, removed and accepted features instead of a single one with a `Status` item to tell wether a feature is active, removed, or accepted. This is mainly due to the way I implemented my macro in the first time and I can switch back to a single array if needed. But an advantage of the way it is now is that when an active feature is used, the parser only searches through the list of active features. It goes through the other arrays only if the feature is not found. I like to think that error checking (in this case, checking that an used feature is active) does not slow down compilation of valid code. :) But this is not very important...
* Feature-gate checking pass now use the `Features` structure instead of looking through a string vector. This should speed them up a bit. The construction of the `Features` struct should be faster too since it is build directly when parsing features instead of calling `has_feature` dozens of times.
* The MacroVisitor pass has been removed, it was mostly useless since the `#[cfg]-stripping` phase happens before (fixes#32648). The features that must actually be checked before expansion are now checked at the time they are used. This also allows us to check attributes that are generated by macro expansion and not visible to MacroVisitor, but are also removed by macro expansion and thus not visible to PostExpansionVisitor either. This fixes#32782. Note that in order for `#[derive_*]` to be feature-gated but still accepted when generated by `#[derive(Trait)]`, I had to do a little bit of trickery with spans that I'm not totally confident into. Please review that part carefully. (It's in `libsyntax_ext/deriving/mod.rs`.)::
Note: this is a [breaking change], since programs with feature-gated attributes on macro-generated macro invocations were not rejected before. For example:
```rust
macro_rules! bar (
() => ()
);
macro_rules! foo (
() => (
#[allow_internal_unstable] //~ ERROR allow_internal_unstable side-steps
bar!();
);
);
```
foo!();
Paths are mostly parsed without taking whitespaces into account, e.g. `std :: vec :: Vec :: new ()` parses successfully, however, there are some special cases involving keywords `super`, `self` and `Self`. For example, `self::` is considered a path start only if there are no spaces between `self` and `::`. These restrictions probably made sense when `self` and friends weren't keywords, but now they are unnecessary.
The first two commits remove this special treatment of whitespaces by removing `token::IdentStyle` entirely and therefore fix https://github.com/rust-lang/rust/issues/14109.
This change also affects naked `self` and `super` (which are not tightly followed by `::`, obviously) they can now be parsed as paths, however they are still not resolved correctly in imports (cc @jseyfried, see `compile-fail/use-keyword.rs`), so https://github.com/rust-lang/rust/issues/29036 is not completely fixed.
The third commit also makes `super`, `self`, `Self` and `static` keywords nominally (before this they acted as keywords for all purposes) and removes most of remaining \"special idents\".
The last commit (before tests) contains some small improvements - some qualified paths with type parameters are parsed correctly, `parse_path` is not used for parsing single identifiers, imports are sanity checked for absence of type parameters - such type parameters can be generated by syntax extensions or by macros when https://github.com/rust-lang/rust/issues/10415 is fixed (~~soon!~~already!).
This patch changes some pretty basic things in `libsyntax`, like `token::Token` and the keyword list, so it's a plugin-[breaking-change].
r? @eddyb
syntax: Merge PathParsingMode::NoTypesAllowed and PathParsingMode::ImportPrefix
syntax: Rename PathParsingMode and its variants to better express their purpose
syntax: Remove obsolete error message about 'self lifetime
syntax: Remove ALLOW_MODULE_PATHS workaround
syntax/resolve: Adjust some error messages
resolve: Compare unhygienic (not renamed) names with keywords::Invalid, invalid identifiers may appear to be valid after renaming
This pass was supposed to check use of gated features before
`#[cfg]`-stripping but this was not the case since it in fact happens
after. Checks that are actually important and must be done before macro
expansion are now made where the features are actually used. Close#32648.
Also ensure that attributes on macro-generated macro invocations are
checked as well. Close#32782 and #32655.
Automated conversion using the untry tool [1] and the following command:
```
$ find -name '*.rs' -type f | xargs untry
```
at the root of the Rust repo.
[1]: https://github.com/japaric/untry
See RFC amendment 1384 and tracking issue 30450:
https://github.com/rust-lang/rfcs/pull/1384https://github.com/rust-lang/rust/issues/30450
Moved old check_matcher code into check_matcher_old
combined the two checks to enable a warning cycle (where we will
continue to error if the two checks agree to reject, accept if the new
check says accept, and warn if the old check accepts but the new check
rejects).
The motivation (other than removing boilerplate) is that this is a baby step towards a parser with error recovery.
[breaking-change] if you use any of the changed functions, you'll need to remove a try! or panictry!
This PR reverts #29543 and instead implements proper support for "=*m" and "+*m" indirect output operands. This provides a framework on top of which support for plain memory operands ("m", "=m" and "+m") can be implemented.
This also fixes the liveness analysis pass not handling read/write operands correctly.
This commit is the standard API stabilization commit for the 1.6 release cycle.
The list of issues and APIs below have all been through their cycle-long FCP and
the libs team decisions are listed below
Stabilized APIs
* `Read::read_exact`
* `ErrorKind::UnexpectedEof` (renamed from `UnexpectedEOF`)
* libcore -- this was a bit of a nuanced stabilization, the crate itself is now
marked as `#[stable]` and the methods appearing via traits for primitives like
`char` and `str` are now also marked as stable. Note that the extension traits
themeselves are marked as unstable as they're imported via the prelude. The
`try!` macro was also moved from the standard library into libcore to have the
same interface. Otherwise the functions all have copied stability from the
standard library now.
* The `#![no_std]` attribute
* `fs::DirBuilder`
* `fs::DirBuilder::new`
* `fs::DirBuilder::recursive`
* `fs::DirBuilder::create`
* `os::unix::fs::DirBuilderExt`
* `os::unix::fs::DirBuilderExt::mode`
* `vec::Drain`
* `vec::Vec::drain`
* `string::Drain`
* `string::String::drain`
* `vec_deque::Drain`
* `vec_deque::VecDeque::drain`
* `collections::hash_map::Drain`
* `collections::hash_map::HashMap::drain`
* `collections::hash_set::Drain`
* `collections::hash_set::HashSet::drain`
* `collections::binary_heap::Drain`
* `collections::binary_heap::BinaryHeap::drain`
* `Vec::extend_from_slice` (renamed from `push_all`)
* `Mutex::get_mut`
* `Mutex::into_inner`
* `RwLock::get_mut`
* `RwLock::into_inner`
* `Iterator::min_by_key` (renamed from `min_by`)
* `Iterator::max_by_key` (renamed from `max_by`)
Deprecated APIs
* `ErrorKind::UnexpectedEOF` (renamed to `UnexpectedEof`)
* `OsString::from_bytes`
* `OsStr::to_cstring`
* `OsStr::to_bytes`
* `fs::walk_dir` and `fs::WalkDir`
* `path::Components::peek`
* `slice::bytes::MutableByteVector`
* `slice::bytes::copy_memory`
* `Vec::push_all` (renamed to `extend_from_slice`)
* `Duration::span`
* `IpAddr`
* `SocketAddr::ip`
* `Read::tee`
* `io::Tee`
* `Write::broadcast`
* `io::Broadcast`
* `Iterator::min_by` (renamed to `min_by_key`)
* `Iterator::max_by` (renamed to `max_by_key`)
* `net::lookup_addr`
New APIs (still unstable)
* `<[T]>::sort_by_key` (added to mirror `min_by_key`)
Closes#27585Closes#27704Closes#27707Closes#27710Closes#27711Closes#27727Closes#27740Closes#27744Closes#27799Closes#27801
cc #27801 (doesn't close as `Chars` is still unstable)
Closes#28968
