rework the README.md for rustc and add other readmes
OK, so, long ago I committed to the idea of trying to write some high-level documentation for rustc. This has proved to be much harder for me to get done than I thought it would! This PR is far from as complete as I had hoped, but I wanted to open it so that people can give me feedback on the conventions that it establishes. If this seems like a good way forward, we can land it and I will open an issue with a good check-list of things to write (and try to take down some of them myself).
Here are the conventions I established on which I would like feedback.
**Use README.md files**. First off, I'm aiming to keep most of the high-level docs in `README.md` files, rather than entries on forge. My thought is that such files are (a) more discoverable than forge and (b) closer to the code, and hence can be edited in a single PR. However, since they are not *in the code*, they will naturally get out of date, so the intention is to focus on the highest-level details, which are least likely to bitrot. I've included a few examples of common functions and so forth, but never tried to (e.g.) exhaustively list the names of functions and so forth.
- I would like to use the tidy scripts to try and check that these do not go out of date. Future work.
**librustc/README.md as the main entrypoint.** This seems like the most natural place people will look first. It lays out how the crates are structured and **is intended** to give pointers to the main data structures of the compiler (I didn't update that yet; the existing material is terribly dated).
**A glossary listing abbreviations and things.** It's much harder to read code if you don't know what some obscure set of letters like `infcx` stands for.
**Major modules each have their own README.md that documents the high-level idea.** For example, I wrote some stuff about `hir` and `ty`. Both of them have many missing topics, but I think that is roughly the level of depth that would be good. The idea is to give people a "feeling" for what the code does.
What is missing primarily here is lots of content. =) Here are some things I'd like to see:
- A description of what a QUERY is and how to define one
- Some comments for `librustc/ty/maps.rs`
- An overview of how compilation proceeds now (i.e., the hybrid demand-driven and forward model) and how we would like to see it going in the future (all demand-driven)
- Some coverage of how incremental will work under red-green
- An updated list of the major IRs in use of the compiler (AST, HIR, TypeckTables, MIR) and major bits of interesting code (typeck, borrowck, etc)
- More advice on how to use `x.py`, or at least pointers to that
- Good choice for `config.toml`
- How to use `RUST_LOG` and other debugging flags (e.g., `-Zverbose`, `-Ztreat-err-as-bug`)
- Helpful conventions for `debug!` statement formatting
cc @rust-lang/compiler @mgattozzi
rustc: Forbid interpolated tokens in the HIR
Right now the HIR contains raw `syntax::ast::Attribute` structure but nowadays
these can contain arbitrary tokens. One variant of the `Token` enum is an
"interpolated" token which basically means to shove all the tokens for a
nonterminal in this position. A "nonterminal" in this case is roughly analagous
to a macro argument:
macro_rules! foo {
($a:expr) => {
// $a is a nonterminal as an expression
}
}
Currently nonterminals contain namely items and expressions, and this poses a
problem for incremental compilation! With incremental we want a stable hash of
all HIR items, but this means we may transitively need a stable hash *of the
entire AST*, which is certainly not stable w/ node ids and whatnot. Hence today
there's a "bug" where the "stable hash" of an AST is just the raw hash value of
the AST, and this only arises with interpolated nonterminals. The downside of
this approach, however, is that a bunch of errors get spewed out during
compilation about how this isn't a great idea.
This PR is focused at fixing these warnings, basically deleting them from the
compiler. The implementation here is to alter attributes as they're lowered from
the AST to HIR, expanding all nonterminals in-place as we see them. This code
for expanding a nonterminal to a token stream already exists for the
`proc_macro` crate, so we basically just reuse the same implementation there.
After this PR it's considered a bug to have an `Interpolated` token and hence
the stable hash implementation simply uses `bug!` in this location.
Closes#40946
hide internal types/traits from std docs via new #[doc(masked)] attribute
Fixes#43701 (hopefully for good this time)
This PR introduces a new parameter to the `#[doc]` attribute that rustdoc looks for on `extern crate` statements. When it sees `#[doc(masked)]` on such a statement, it hides traits and types from that crate from appearing in either the "Trait Implementations" section of many type pages, or the "Implementors" section of trait pages. This is then applied to the `libc`/`rand`/`compiler_builtins` imports in libstd to prevent those crates from creating broken links in the std docs.
Like in #43348, this also introduces a feature gate, `doc_masked`, that controls the use of this parameter.
To view the std docs generated with this change, head to https://tonberry.quietmisdreavus.net/std-43701/std/index.html.
Right now the HIR contains raw `syntax::ast::Attribute` structure but nowadays
these can contain arbitrary tokens. One variant of the `Token` enum is an
"interpolated" token which basically means to shove all the tokens for a
nonterminal in this position. A "nonterminal" in this case is roughly analagous
to a macro argument:
macro_rules! foo {
($a:expr) => {
// $a is a nonterminal as an expression
}
}
Currently nonterminals contain namely items and expressions, and this poses a
problem for incremental compilation! With incremental we want a stable hash of
all HIR items, but this means we may transitively need a stable hash *of the
entire AST*, which is certainly not stable w/ node ids and whatnot. Hence today
there's a "bug" where the "stable hash" of an AST is just the raw hash value of
the AST, and this only arises with interpolated nonterminals. The downside of
this approach, however, is that a bunch of errors get spewed out during
compilation about how this isn't a great idea.
This PR is focused at fixing these warnings, basically deleting them from the
compiler. The implementation here is to alter attributes as they're lowered from
the AST to HIR, expanding all nonterminals in-place as we see them. This code
for expanding a nonterminal to a token stream already exists for the
`proc_macro` crate, so we basically just reuse the same implementation there.
After this PR it's considered a bug to have an `Interpolated` token and hence
the stable hash implementation simply uses `bug!` in this location.
Closes#40946
incr.comp.: Compute fingerprint for all query results.
This PR enables query result fingerprinting in incremental mode. This is an essential piece of infrastructure for red/green tracking. We don't do anything with the fingerprints yet but merging the infrastructure should protect it from bit-rotting and will make it easier to start measuring its performance impact (and thus let us determine if we should switch to a faster hashing algorithm rather sooner than later).
Note, this PR also includes the changes from https://github.com/rust-lang/rust/pull/43887 which I'm therefore closing. No need to re-review the first commit though.
r? @nikomatsakis
Otherwise we may emit double errors related to the `#[export_name]` attribute,
for example, and using a query should ensure that it's only emitted at most
once.
Individualize feature gates for const fn invocation
This PR changes the meaning of `#![feature(const_fn)]` so it is only required to declare a const fn but not to call one. Based on discussion at #24111. I was hoping we could have an FCP here in order to move that conversation forward.
This sets the stage for future stabilization of the constness of several functions in the standard library (listed below), so could someone please tag the lang team for review.
- `std::cell`
- `Cell::new`
- `RefCell::new`
- `UnsafeCell::new`
- `std::mem`
- `size_of`
- `align_of`
- `std::ptr`
- `null`
- `null_mut`
- `std::sync`
- `atomic`
- `Atomic{Bool,Ptr,Isize,Usize}::new`
- `once`
- `Once::new`
- primitives
- `{integer}::min_value`
- `{integer}::max_value`
Some other functions are const but they are also unstable or hidden, e.g. `Unique::new` so they don't have to be considered at this time.
After this stabilization, the following `*_INIT` constants in the standard library can be deprecated. I wasn't sure whether to include those deprecations in the current PR.
- `std::sync`
- `atomic`
- `ATOMIC_{BOOL,ISIZE,USIZE}_INIT`
- `once`
- `ONCE_INIT`
Parse nested closure with two consecutive parameter lists properly
This is a followup of #44332.
---
Currently, in nightly, this does not compile:
```rust
fn main() {
let f = |_||x, y| x+y;
println!("{}", f(())(1, 2)); // should print 3
}
```
`|_||x, y| x+y` should be parsed as `|_| (|x, y| x+y)`, but the parser didn't accept `||` between `_` and `x`. This patch fixes the problem.
r? @petrochenkov
Expect pipe symbol after closure parameter list
Fixes#44021.
---
Originally, the parser just called `bump` to discard following token after parsing closure parameter list, because it assumes `|` is following. However, the following code breaks the assumption:
```rust
struct MyStruct;
impl MyStruct {
fn f() {|x, y}
}
```
Here, the parameter list is `x, y` and the following token is `}`. The parser discards `}`, and then we have a curly bracket mismatch.
Indeed, this code has a syntax error. On current nightly, the compiler emits an syntax error, but with incorrect message and span, followed by an ICE.
```
error: expected expression, found `}`
--> 44021.rs:4:1
|
4 | }
| ^
error: internal compiler error: unexpected panic
```
Even worse, on current stable(1.20.0), the compiler falls into an infinite loop.
This pull request fixes this problem. Now the compiler emits correct error message and span, and does not ICE.
```
error: expected one of `:`, `@`, or `|`, found `}`
--> 44021.rs:3:20
|
3 | fn foo() {|x, y}
| ^ expected one of `:`, `@`, or `|` here
```
rustc: Separately feature gate repr(i128)
Brought up during the discussion of #35118, the support for this is still
somewhat buggy and so stabilization likely wants to be considered independently
of the type itself.
pprust: fix parenthesization of exprs
The pretty printer in `syntax::print::pprust` currently relies on the presence of `ExprKind::Paren` hints in order to correctly parenthesize expressions in its output. If `Paren` nodes are missing, it sometimes produces wrong output, such as printing `1 - (2 - 3)` as `1 - 2 - 3`. This PR fixes `pprust` to correctly print expressions regardless of the presence or absence of `Paren` nodes. This should make `pprust` easier to use with programmatically constructed ASTs.
A few notes:
* I added a function for assigning precedence values to exprs in `syntax::util::parser`, since there is already code there for assigning precedence values to binops. Let me know if I should move this somewhere more `pprust`-specific.
* I also moved the `contains_exterior_struct_lit` function from `rustc_lint::unused::UnusedParens` into `syntax::util::parser`, since it's needed for determining the correct parenthesization of `if`/`while` conditional expressions.
* I couldn't find a good way to compare two exprs for equivalence while ignoring semantically-irrelevant details like spans. So the test for the new behavior relies on a slight hack: it adds `Paren` nodes everywhere, so that the pretty-printed version exactly reflects the structure of the AST, and then compares the printed strings. This works, but let me know if there's a better way.
Brought up during the discussion of #35118, the support for this is still
somewhat buggy and so stabilization likely wants to be considered independently
of the type itself.
feature error span on attribute for fn_must_use, SIMD/align reprs, macro reëxport
There were several feature-gated attributes for which the feature-not-available
error spans would point to the item annotated with the gated attribute, when it
would make more sense for the span to point to the attribute itself: if the
attribute is removed, the function/struct/_&c._ likely still makes sense and the
program will compile. (Note that we decline to make the analogous change for
the `main`, `start`, and `plugin_registrar` features, for in those cases it
makes sense for the span to implicate the entire function, of which there is
little hope of using without the gated attribute.)
There were several feature-gated attributes for which the
feature-not-available error spans would point to the item annotated with
the gated attribute, when it would make more sense for the span to point
to the attribute itself: if the attribute is removed, the
function/struct/&c. likely still makes sense and the program will
compile. (Note that we decline to make the analogous change for the
`main`, `start`, and `plugin_registrar` features, for in those cases it
makes sense for the span to implicate the entire function, of which
there is little hope of using without the gated attribute.)
