This commit moves compression of the bytecode from the `link` module to the
`write` module, namely allowing it to be (a) cached by incremental compilation
and (b) produced in parallel. The parallelization may show up as some nice wins
during normal compilation and the caching in incremental mode should be
beneficial for incremental compiles! (no more need to recompress the entire
crate's bitcode on all builds)
This commit is a refactoring of the LTO backend in Rust to support compilations
with multiple codegen units. The immediate result of this PR is to remove the
artificial error emitted by rustc about `-C lto -C codegen-units-8`, but longer
term this is intended to lay the groundwork for LTO with incremental compilation
and ultimately be the underpinning of ThinLTO support.
The problem here that needed solving is that when rustc is producing multiple
codegen units in one compilation LTO needs to merge them all together.
Previously only upstream dependencies were merged and it was inherently relied
on that there was only one local codegen unit. Supporting this involved
refactoring the optimization backend architecture for rustc, namely splitting
the `optimize_and_codegen` function into `optimize` and `codegen`. After an LLVM
module has been optimized it may be blocked and queued up for LTO, and only
after LTO are modules code generated.
Non-LTO compilations should look the same as they do today backend-wise, we'll
spin up a thread for each codegen unit and optimize/codegen in that thread. LTO
compilations will, however, send the LLVM module back to the coordinator thread
once optimizations have finished. When all LLVM modules have finished optimizing
the coordinator will invoke the LTO backend, producing a further list of LLVM
modules. Currently this is always a list of one LLVM module. The coordinator
then spawns further work to run LTO and code generation passes over each module.
In the course of this refactoring a number of other pieces were refactored:
* Management of the bytecode encoding in rlibs was centralized into one module
instead of being scattered across LTO and linking.
* Some internal refactorings on the link stage of the compiler was done to work
directly from `CompiledModule` structures instead of lists of paths.
* The trans time-graph output was tweaked a little to include a name on each
bar and inflate the size of the bars a little
This commit removes the `crate_trans_items` field from the `CrateContext` of
trans. This field, a big map, was calculated during partioning and was a set of
all translation items. This isn't quite incremental-friendly because the map may
change a lot but not have much effect on downstream consumers.
Instead a new query was added for the one location this map was needed, along
with a new comment explaining what the location is doing!
This is a big map that ends up inside of a `CrateContext` during translation for
all codegen units. This means that any change to the map may end up causing an
incremental recompilation of a codegen unit! In order to reduce the amount of
dependencies here between codegen units and the actual input crate this commit
refactors dealing with exported symbols and such into various queries.
The new queries are largely based on existing queries with filled out
implementations for the local crate in addition to external crates, but the main
idea is that while translating codegen untis no unit needs the entire set of
exported symbols, instead they only need queries about particulare `DefId`
instances every now and then.
The linking stage, however, still generates a full list of all exported symbols
from all crates, but that's going to always happen unconditionally anyway, so no
news there!
Otherwise we may emit double errors related to the `#[export_name]` attribute,
for example, and using a query should ensure that it's only emitted at most
once.
This commit moves the `collect_and_partition_translation_items` function into a
query on `TyCtxt` instead of a free function in trans, allowing us to track
dependencies and such of the function.
This commit moves the definition of the `ExportedSymbols` structure to the
`rustc` crate and then creates a query that'll be used to construct the
`ExportedSymbols` set. This in turn uses the reachablity query exposed in the
previous commit.
Migrate a slew of metadata methods to queries
This PR intends to make more progress on #41417, knocking off some low-hanging fruit.
Closes#44190
cc #44137
This commit adds logic to the compiler to attempt to handle super long linker
invocations by falling back to the `@`-file syntax if the invoked command is too
large. Each OS has a limit on how many arguments and how large the arguments can
be when spawning a new process, and linkers tend to be one of those programs
that can hit the limit!
The logic implemented here is to unconditionally attempt to spawn a linker and
then if it fails to spawn with an error from the OS that indicates the command
line is too big we attempt a fallback. The fallback is roughly the same for all
linkers where an argument pointing to a file, prepended with `@`, is passed.
This file then contains all the various arguments that we want to pass to the
linker.
Closes#41190
The main use of `CrateStore` *before* the `TyCtxt` is created is during
resolution, but we want to be sure that any methods used before resolution are
not used after the `TyCtxt` is created. This commit starts moving the methods
used by resolve to all be named `{name}_untracked` where the rest of the
compiler uses just `{name}` as a query.
During this transition a number of new queries were added to account for
post-resolve usage of these methods.
This commit started by moving methods from `CrateStore` to queries, but it ended
up necessitating some deeper refactorings to move more items in general to
queries.
Before this commit the *resolver* would walk over the AST and process foreign
modules (`extern { .. }` blocks) and collect `#[link]` annotations. It would
then also process the command line `-l` directives and such. This information
was then stored as precalculated lists in the `CrateStore` object for iterating
over later.
After this, commit, however, this pass no longer happens during resolution but
now instead happens through queries. A query for the linked libraries of a crate
will crawl the crate for `extern` blocks and then process the linkage
annotations at that time.
This comit applies the following changes:
* Deletes the `is_allocator` query as it's no longer used
* Moves the `is_sanitizer_runtime` method to a query
* Moves the `is_profiler_runtime` method to a query
* Moves the `panic_strategy` method to a query
* Moves the `is_no_builtins` method to a query
* Deletes the cstore method of `is_compiler_builtins`. The query was added in
#42588 but the `CrateStore` method was not deleted
A good bit of these methods were used late in linking during trans so a new
dedicated structure was created to ship a calculated form of this information
over to the linker rather than having to ship the whole of `TyCtxt` over to
linking.
This PR is an implementation of [RFC 1974] which specifies a new method of
defining a global allocator for a program. This obsoletes the old
`#![allocator]` attribute and also removes support for it.
[RFC 1974]: https://github.com/rust-lang/rfcs/pull/197
The new `#[global_allocator]` attribute solves many issues encountered with the
`#![allocator]` attribute such as composition and restrictions on the crate
graph itself. The compiler now has much more control over the ABI of the
allocator and how it's implemented, allowing much more freedom in terms of how
this feature is implemented.
cc #27389
This commit integrates the `jobserver` crate into the compiler. The crate was
previously integrated in to Cargo as part of rust-lang/cargo#4110. The purpose
here is to two-fold:
* Primarily the compiler can cooperate with Cargo on parallelism. When you run
`cargo build -j4` then this'll make sure that the entire build process between
Cargo/rustc won't use more than 4 cores, whereas today you'd get 4 rustc
instances which may all try to spawn lots of threads.
* Secondarily rustc/Cargo can now integrate with a foreign GNU `make` jobserver.
This means that if you call cargo/rustc from `make` or another
jobserver-compatible implementation it'll use foreign parallelism settings
instead of creating new ones locally.
As the number of parallel codegen instances in the compiler continues to grow
over time with the advent of incremental compilation it's expected that this'll
become more of a problem, so this is intended to nip concurrent concerns in the
bud by having all the tools to cooperate!
Note that while rustc has support for itself creating a jobserver it's far more
likely that rustc will always use the jobserver configured by Cargo. Cargo today
will now set a jobserver unconditionally for rustc to use.
A long time coming this commit removes the `flate` crate in favor of the
`flate2` crate on crates.io. The functionality in `flate2` originally flowered
out of `flate` itself and is additionally the namesake for the crate. This will
leave a gap in the naming (there's not `flate` crate), which will likely cause a
particle collapse of some form somewhere.
This commit
* Refactors the collect_lib_features function to work in a
non-checking mode (no bad pointer needed, and list of
lang features).
* Introduces checking whether unstable/stable tags for a
given feature have inconsistent tracking issues.
* Fixes such inconsistencies throughout the codebase.
Mark various items and fields as private or pub(crate), and remove a function that turns out to be unused.
These are not used anywhere in-tree, but I guess it's a [breaking-change] for plugins.
This does not actually improve build times, since it still depends
on rustc_trans, but is better layering and fits the multi-backend
future slightly better.
Consequently, session creation can no longer initialize LLVM.
The few places that use the compiler without going through
rustc_driver/CompilerCalls thus need to be careful to manually
initialize LLVM (via rustc_trans!) immediately after session
creation.
This means librustc is not rebuilt when LLVM changes.
Move the code for loading metadata from rlibs and dylibs from
rustc_metadata into rustc_trans, and introduce a trait to avoid
introducing a direct dependency on rustc_trans.
This means rustc_metadata is no longer rebuilt when LLVM changes.
Removes unused macros from:
* libcore
* libcollections
The last use of these two macros was removed in commit
b64c9d56700e2c41207166fe8709711ff02488ff
when the char_range_at_reverse function was been removed.
* librustc_errors
Their last use was removed by commits
2f2c3e178325dc1837badcd7573c2c0905fab979
and 11dc974a38fd533aa692cea213305056cd3a6902.
* libsyntax_ext
* librustc_trans
Also, put the otry macro in back/msvc/mod.rs under the
same cfg argument as the places that use it.
