e8c337b5ca
This is roughly the same as my previous PR that created a dependency graph, but that: 1. The dependency graph is only optionally constructed, though this doesn't seem to make much of a difference in terms of overhead (see measurements below). 2. The dependency graph is simpler (I combined a lot of nodes). 3. The dependency graph debugging facilities are much better: you can now use `RUST_DEP_GRAPH_FILTER` to filter the dep graph to just the nodes you are interested in, which is super help. 4. The tests are somewhat more elaborate, including a few known bugs I need to fix in a second pass. This is potentially a `[breaking-change]` for plugin authors. If you are poking about in tcx state or something like that, you probably want to add `let _ignore = tcx.dep_graph.in_ignore();`, which will cause your reads/writes to be ignored and not affect the dep-graph. After this, or perhaps as an add-on to this PR in some cases, what I would like to do is the following: - [x] Write-up a little guide to how to use this system, the debugging options available, and what the possible failure modes are. - [ ] Introduce read-only and perhaps the `Meta` node - [x] Replace "memoization tasks" with node from the map itself - [ ] Fix the shortcomings, obviously! Notably, the HIR map needs to register reads, and there is some state that is not yet tracked. (Maybe as a separate PR.) - [x] Refactor the dep-graph code so that the actual maintenance of the dep-graph occurs in a parallel thread, and the main thread simply throws things into a shared channel (probably a fixed-size channel). There is no reason for dep-graph construction to be on the main thread. (Maybe as a separate PR.) Regarding performance: adding this tracking does add some overhead, approximately 2% in my measurements (I was comparing the build times for rustdoc). Interestingly, enabling or disabling tracking doesn't seem to do very much. I want to poke at this some more and gather a bit more data -- in some tests I've seen that 2% go away, but on others it comes back. It's not entirely clear to me if that 2% is truly due to constructing the dep-graph at all. The next big step after this is write some code to dump the dep-graph to disk and reload it. r? @michaelwoerister |
||
---|---|---|
.. | ||
dep_graph | ||
front | ||
lint | ||
middle | ||
mir | ||
session | ||
util | ||
diagnostics.rs | ||
lib.rs | ||
macros.rs | ||
README.md |
An informal guide to reading and working on the rustc compiler.
If you wish to expand on this document, or have a more experienced Rust contributor add anything else to it, please get in touch:
or file a bug:
https://github.com/rust-lang/rust/issues
Your concerns are probably the same as someone else's.
The crates of rustc
Rustc consists of a number of crates, including libsyntax
,
librustc
, librustc_back
, librustc_trans
, and librustc_driver
(the names and divisions are not set in stone and may change;
in general, a finer-grained division of crates is preferable):
-
libsyntax
contains those things concerned purely with syntax – that is, the AST, parser, pretty-printer, lexer, macro expander, and utilities for traversing ASTs – are in a separate crate called "syntax", whose files are in./../libsyntax
, where.
is the current directory (that is, the parent directory of front/, middle/, back/, and so on). -
librustc
(the current directory) contains the high-level analysis passes, such as the type checker, borrow checker, and so forth. It is the heart of the compiler. -
librustc_back
contains some very low-level details that are specific to different LLVM targets and so forth. -
librustc_trans
contains the code to convert from Rust IR into LLVM IR, and then from LLVM IR into machine code, as well as the main driver that orchestrates all the other passes and various other bits of miscellany. In general it contains code that runs towards the end of the compilation process. -
librustc_driver
invokes the compiler fromlibsyntax
, then the analysis phases fromlibrustc
, and finally the lowering and codegen passes fromlibrustc_trans
.
Roughly speaking the "order" of the three crates is as follows:
libsyntax -> librustc -> librustc_trans
| |
+-----------------+-------------------+
|
librustc_driver
Modules in the rustc crate
The rustc crate itself consists of the following submodules (mostly, but not entirely, in their own directories):
- session: options and data that pertain to the compilation session as a whole
- middle: middle-end: name resolution, typechecking, LLVM code generation
- metadata: encoder and decoder for data required by separate compilation
- plugin: infrastructure for compiler plugins
- lint: infrastructure for compiler warnings
- util: ubiquitous types and helper functions
- lib: bindings to LLVM
The entry-point for the compiler is main() in the librustc_driver crate.
The 3 central data structures:
-
./../libsyntax/ast.rs
defines the AST. The AST is treated as immutable after parsing, but it depends on mutable context data structures (mainly hash maps) to give it meaning.-
Many – though not all – nodes within this data structure are wrapped in the type
spanned<T>
, meaning that the front-end has marked the input coordinates of that node. The membernode
is the data itself, the memberspan
is the input location (file, line, column; both low and high). -
Many other nodes within this data structure carry a
def_id
. These nodes represent the 'target' of some name reference elsewhere in the tree. When the AST is resolved, bymiddle/resolve.rs
, all names wind up acquiring a def that they point to. So anything that can be pointed-to by a name winds up with adef_id
.
-
-
middle/ty.rs
defines the datatypesty
. This is the type that represents types after they have been resolved and normalized by the middle-end. The typeck phase converts every ast type to aty::sty
, and the latter is used to drive later phases of compilation. Most variants in theast::ty
tag have a corresponding variant in thety::sty
tag. -
./../librustc_llvm/lib.rs
defines the exported typesValueRef
,TypeRef
,BasicBlockRef
, and several others. Each of these is an opaque pointer to an LLVM type, manipulated through thelib::llvm
interface.
Control and information flow within the compiler:
-
main() in lib.rs assumes control on startup. Options are parsed, platform is detected, etc.
-
./../libsyntax/parse/parser.rs
parses the input files and produces an AST that represents the input crate. -
Multiple middle-end passes (
middle/resolve.rs
,middle/typeck.rs
) analyze the semantics of the resulting AST. Each pass generates new information about the AST and stores it in various environment data structures. The driver passes environments to each compiler pass that needs to refer to them. -
Finally, the
trans
module inlibrustc_trans
translates the Rust AST to LLVM bitcode in a type-directed way. When it's finished synthesizing LLVM values, rustc asks LLVM to write them out in some form (.bc
,.o
) and possibly run the system linker.