This allows you to enable *all* nested visits in a future-compatible
sort of way. Moreover, if you choose to override the `visit_nested`
methods yourself, you can "future-proof" against omissions by overriding
`nested_visit_map` to panic.
Before this PR, type names could depend on the cratenum being used
for a given crate and also on the source location of closures.
Both are undesirable for incremental compilation where we cache
LLVM IR and don't want it to depend on formatting or in which
order crates are loaded.
Replace FNV with a faster hash function.
Hash table lookups are very hot in rustc profiles and the time taken within `FnvHash` itself is a big part of that. Although FNV is a simple hash, it processes its input one byte at a time. In contrast, Firefox has a homespun hash function that is also simple but works on multiple bytes at a time. So I tried it out and the results are compelling:
```
futures-rs-test 4.326s vs 4.212s --> 1.027x faster (variance: 1.001x, 1.007x)
helloworld 0.233s vs 0.232s --> 1.004x faster (variance: 1.037x, 1.016x)
html5ever-2016- 5.397s vs 5.210s --> 1.036x faster (variance: 1.009x, 1.006x)
hyper.0.5.0 5.018s vs 4.905s --> 1.023x faster (variance: 1.007x, 1.006x)
inflate-0.1.0 4.889s vs 4.872s --> 1.004x faster (variance: 1.012x, 1.007x)
issue-32062-equ 0.347s vs 0.335s --> 1.035x faster (variance: 1.033x, 1.019x)
issue-32278-big 1.717s vs 1.622s --> 1.059x faster (variance: 1.027x, 1.028x)
jld-day15-parse 1.537s vs 1.459s --> 1.054x faster (variance: 1.005x, 1.003x)
piston-image-0. 11.863s vs 11.482s --> 1.033x faster (variance: 1.060x, 1.002x)
regex.0.1.30 2.517s vs 2.453s --> 1.026x faster (variance: 1.011x, 1.013x)
rust-encoding-0 2.080s vs 2.047s --> 1.016x faster (variance: 1.005x, 1.005x)
syntex-0.42.2 32.268s vs 31.275s --> 1.032x faster (variance: 1.014x, 1.022x)
syntex-0.42.2-i 17.629s vs 16.559s --> 1.065x faster (variance: 1.013x, 1.021x)
```
(That's a stage1 compiler doing debug builds. Results for a stage2 compiler are similar.)
The attached commit is not in a state suitable for landing because I changed the implementation of FnvHasher without changing its name (because that would have required touching many lines in the compiler). Nonetheless, it is a good place to start discussions.
Profiles show very clearly that this new hash function is a lot faster to compute than FNV. The quality of the new hash function is less clear -- it seems to do better in some cases and worse in others (judging by the number of instructions executed in `Hash{Map,Set}::get`).
CC @brson, @arthurprs
Stabilize `..` in tuple (struct) patterns
I'd like to nominate `..` in tuple and tuple struct patterns for stabilization.
This feature is a relatively small extension to existing stable functionality and doesn't have known blockers.
The feature first appeared in Rust 1.10 6 months ago.
An example of use: https://github.com/rust-lang/rust/pull/36203
Closes https://github.com/rust-lang/rust/issues/33627
r? @nikomatsakis
Reduce the number of bytes hashed by IchHasher.
IchHasher uses blake2b hashing, which is expensive, so the fewer bytes hashed
the better. There are two big ways to reduce the number of bytes hashed.
- Filenames in spans account for ~66% of all bytes (for builds with debuginfo).
The vast majority of spans have the same filename for the start of the span
and the end of the span, so hashing the filename just once in those cases is
a big win.
- u32 and u64 and usize values account for ~25%--33% of all bytes (for builds
with debuginfo). The vast majority of these are small, i.e. fit in a u8, so
shrinking them down before hashing is also a big win.
This PR implements these two optimizations. I'm certain the first one is safe.
I'm about 90% sure that the second one is safe.
Here are measurements of the number of bytes hashed when doing
debuginfo-enabled builds of stdlib and
rustc-benchmarks/syntex-0.42.2-incr-clean.
```
stdlib syntex-incr
------ -----------
original 156,781,386 255,095,596
half-SawSpan 106,744,403 176,345,419
short-ints 45,890,534 118,014,227
no-SawSpan[*] 6,831,874 45,875,714
[*] don't hash the SawSpan at all. Not part of this PR, just implemented for
comparison's sake.
```
For debug builds of syntex-0.42.2-incr-clean, the two changes give a 1--2%
speed-up.
ICH: Hash expression spans if their source location is captured for panics.
Since the location of some expressions is captured in error message constants, it has an influence on machine code and consequently we need to take them into account by the incr. comp. hash. This PR makes this happen for `+, -, *, /, %` and for array indexing -- let me know if I forgot anything.
In the future we might want to change the codegen strategy for those error messages, so that they are stored in a separate object file with a stable symbol name, so that only this object file has to be regenerated when source locations change. This strategy would also eliminate unnecessary duplications due to monomorphization, as @arielb1 has pointed out on IRC. I opened https://github.com/rust-lang/rust/issues/37512, so we don't forget about this.
r? @nikomatsakis
std: Stabilize and deprecate APIs for 1.13
This commit is intended to be backported to the 1.13 branch, and works with the
following APIs:
Stabilized
* `i32::checked_abs`
* `i32::wrapping_abs`
* `i32::overflowing_abs`
* `RefCell::try_borrow`
* `RefCell::try_borrow_mut`
Deprecated
* `BinaryHeap::push_pop`
* `BinaryHeap::replace`
* `SipHash13`
* `SipHash24`
* `SipHasher` - use `DefaultHasher` instead in the `std::collections::hash_map`
module
Closes#28147Closes#34767Closes#35057Closes#35070
This commit is intended to be backported to the 1.13 branch, and works with the
following APIs:
Stabilized
* `i32::checked_abs`
* `i32::wrapping_abs`
* `i32::overflowing_abs`
* `RefCell::try_borrow`
* `RefCell::try_borrow_mut`
* `DefaultHasher`
* `DefaultHasher::new`
* `DefaultHasher::default`
Deprecated
* `BinaryHeap::push_pop`
* `BinaryHeap::replace`
* `SipHash13`
* `SipHash24`
* `SipHasher` - use `DefaultHasher` instead in the `std::collections::hash_map`
module
Closes#28147Closes#34767Closes#35057Closes#35070
fix a few errant `Krate` edges
Exploring the effect of small changes on `syntex` reuse, I discovered the following sources of unnecessary edges from `Krate`
r? @michaelwoerister
incr. comp.: Take spans into account for ICH
This PR makes the ICH (incr. comp. hash) take spans into account when debuginfo is enabled.
A side-effect of this is that the SVH (which is based on the ICHs of all items in the crate) becomes sensitive to the tiniest change in a code base if debuginfo is enabled. Since we are not trying to model ABI compatibility via the SVH anymore (this is done via the crate disambiguator now), this should be not be a problem.
Fixes#33888.
Fixes#32753.
Implement synchronization scheme for incr. comp. directory
This PR implements a copy-on-write-based synchronization scheme for the incremental compilation cache directory. For technical details, see the documentation at the beginning of `rustc_incremental/persist/fs.rs`.
The PR contains unit tests for some functions but for testing whether the scheme properly handles races, a more elaborate test setup would be needed. It would probably involve a small tool that allows to manipulate the incremental compilation directory in a controlled way and then letting a compiler instance run against directories in different states. I don't know if it's worth the trouble of adding another test category to `compiletest`, but I'd be happy to do so.
Fixes#32754Fixes#34957
Experimentally, this fixes the poor re-use observed in
libsyntex-syntax. I'm not sure how to make a regression test for this,
though, given the non-deterministic nature of it.
This avoids the compile-time overhead of computing them twice. It also fixes
an issue where the hash computed after typeck is differen than the hash before,
because typeck mutates the def-map in place.
Fixes#35549.
Fixes#35593.
This massively speeds up serialization. It also
seems to produce deterministic metadata hashes
(before I was seeing inconsistent results).
Fixes#35232.
When we hash the inputs to a MetaData node, we have to hash them in a
consistent order. We achieve this by sorting the stringfied `DefPath`
entries. Also, micro-optimie by cache more results across the saving
process.
The biggest problem, actually, is krate numbers being removed entirely,
which can lead to array-index-out-of-bounds errors.
cc #35123 -- not a complete fix, since really we ought to "map" the old
crate numbers to the new ones, not just detect changes.
The way we do HIR inlining introduces reads of the "Hir" into the graph,
but this Hir in fact belongs to other crates, so when we try to load
later, we ICE because the Hir nodes in question don't belond to the
crate (and we haven't done inlining yet). This pass rewrites those HIR
nodes to the metadata from which the inlined HIR was loaded.
Better attribute and metaitem encapsulation throughout the compiler
This PR refactors most (hopefully all?) of the `MetaItem` interactions outside of `libsyntax` (and a few inside) to interact with MetaItems through the provided traits instead of directly creating / destruct / matching against them. This is a necessary first step to eventually converting `MetaItem`s to internally use `TokenStream` representations (which will make `MetaItem` interactions much nicer for macro writers once the new macro system is in place).
r? @nrc
In the older version, a `.o` and ` .bc` file were separate
work-products. This newer version keeps, for each codegen-unit, a set
of files of different kinds. We assume that if any kinds are available
then all the kinds we need are available, since the precise set of
switches will depend on attributes and command-line switches.
Should probably test this: the effect of changing attributes in
particular might not be successfully tracked?
This checks the `previous_work_products` data from the dep-graph and
tries to simply copy a `.o` file if possible. We also add new
work-products into the dep-graph, and create edges to/from the dep-node
for a work-product.
We used to use `Name`, but the session outlives the tokenizer, which
means that attempts to read this field after trans has complete
otherwise panic. All reads want an `InternedString` anyhow.
To handle the general case, we include a vector of def-ids, so that we
can account for things like `(Foo, Bar)` which references both `Foo` and
`Bar`. This means it is not Copy, so re-jigger some APIs to use
borrowing more intelligently.
Apply visit_path to import prefixes by default
Overriding `visit_path` is not enough to visit all paths, some import prefixes are not visited and `visit_path_list_item` need to be overridden as well. This PR removes this catch, it should be less error prone this way. Also, the prefix is visited once now, not repeatedly for each path list item.
r? @eddyb
This makes the "shadowing labels" warning *not* print the entire loop
as a span, but only the lifetime.
Also makes #31719 go away, but does not fix its root cause (the span
of the expanded loop is still wonky, but not used anymore).
This commit reorganizes how the persist code treats hashing. The idea is
that each crate saves a file containing hashes representing the metadata
for each item X. When we see a read from `MetaData(X)`, we can load this
hash up (if we don't find a file for that crate, we just use the SVH for
the entire crate).
To compute the hash for `MetaData(Y)`, where Y is some local item, we
examine all the predecessors of the `MetaData(Y)` node and hash their
hashes together.
For external crates, we must build up a map that goes from
the DefKey to the DefIndex. We do this by iterating over each
index that is found in the metadata and loading the associated
DefKey.
The compiler created a directory as part of `-Z incremental` but that may be
hierarchically used concurrently so we need to protect ourselves against that.
This commit rewrites all of the tidy checks we have, namely:
* featureck
* errorck
* tidy
* binaries
into Rust under a new `tidy` tool inside of the `src/tools` directory. This at
the same time deletes all the corresponding Python tidy checks so we can be sure
to only have one source of truth for all the tidy checks.
cc #31590
