Commit Graph

274 Commits

Author SHA1 Message Date
Camille GILLOT
fe89f3236c Address review. 2021-03-30 18:10:08 +02:00
Camille GILLOT
65a8681a17 Add documentation. 2021-03-30 18:10:07 +02:00
Camille GILLOT
c5c935af92 Simplify tracking the encoder state. 2021-03-30 18:10:07 +02:00
Camille GILLOT
e1c99e5fcc Remove the parallel version. 2021-03-30 18:10:06 +02:00
Camille GILLOT
8208872fa2 Fix parallel compiler. 2021-03-30 18:10:06 +02:00
Camille GILLOT
cfe786e5e0 Fix tests.
Avoid invoking queries inside `check_paths`, since we are holding a lock
to the reconstructed graph.
2021-03-30 18:10:06 +02:00
Camille GILLOT
39b306a53d Do not allocate in decoder. 2021-03-30 18:10:05 +02:00
Camille GILLOT
6bfaf3a9cb Stream the dep-graph to a file. 2021-03-30 18:09:59 +02:00
Joshua Nelson
441dc3640a Remove (lots of) dead code
Found with https://github.com/est31/warnalyzer.

Dubious changes:
- Is anyone else using rustc_apfloat? I feel weird completely deleting
  x87 support.
- Maybe some of the dead code in rustc_data_structures, in case someone
  wants to use it in the future?
- Don't change rustc_serialize

  I plan to scrap most of the json module in the near future (see
  https://github.com/rust-lang/compiler-team/issues/418) and fixing the
  tests needed more work than I expected.

TODO: check if any of the comments on the deleted code should be kept.
2021-03-27 22:16:33 -04:00
Josh Stone
72ebebe474 Use iter::zip in compiler/ 2021-03-26 09:32:31 -07:00
Aaron Hill
443cef5618
Debug-print result when an unstable fingerprint is detected 2021-03-19 21:47:57 -04:00
bors
2a55274e0c Auto merge of #82999 - cuviper:rustc-rayon-0.3.1, r=Mark-Simulacrum
Update to rustc-rayon 0.3.1

This pulls in rust-lang/rustc-rayon#8 to fix #81425. (h/t `@ammaraskar)`

That revealed weak constraints on `rustc_arena::DropArena`, because its
`DropType` was holding type-erased raw pointers to generic `T`. We can
implement `Send` for `DropType` (under `cfg(parallel_compiler)`) by
requiring all `T: Send` before they're type-erased.
2021-03-15 08:49:25 +00:00
bors
e7e1dc158c Auto merge of #83007 - Aaron1011:incr-verify-default, r=Mark-Simulacrum
Turn `-Z incremental-verify-ich` on by default

Issue #82920 showed that the kind of bugs caught by this flag have
soundness implications.
2021-03-13 17:52:22 +00:00
Aaron Hill
7d7c81a114
Always run incremental_verify_ich when re-computing query results
Issue #82920 showed that the kind of bugs caught by this flag have
soundness implications.

This causes performance regressions of up to 15.2% during incremental
compilation, but this is necessary to catch miscompilations caused by
bugs in query implementations.
2021-03-13 12:00:38 -05:00
Tyson Nottingham
adcbe49b16 rustc_query_system: simplify QueryCache::iter
Minor cleanup to reduce a small amount of complexity and code bloat.
Reduces the number of mono items in rustc_query_impl by 15%.
2021-03-12 17:34:14 -08:00
Josh Stone
f7e75a2124 Update to rustc-rayon 0.3.1
This pulls in rust-lang/rustc-rayon#8 to fix #81425. (h/t @ammaraskar)

That revealed weak constraints on `rustc_arena::DropArena`, because its
`DropType` was holding type-erased raw pointers to generic `T`. We can
implement `Send` for `DropType` (under `cfg(parallel_compiler)`) by
requiring all `T: Send` before they're type-erased.
2021-03-10 17:53:35 -08:00
Camille GILLOT
903f65f215 Simplify hashing. 2021-02-21 12:22:22 +01:00
Camille GILLOT
a87de890fd Move print_query_stack to rustc_query_system. 2021-02-20 23:40:56 +01:00
Camille GILLOT
c26d965714 Move report_cycle to rustc_query_system.
The call to `ty::print::with_forced_impl_filename_line`
is done when constructing the description,
at the construction of the QueryStackFrame.
2021-02-20 23:36:31 +01:00
Camille GILLOT
3897395787 Move Query to rustc_query_system.
Rename it to QueryStackFrame and document a bit.
2021-02-20 22:53:47 +01:00
Camille GILLOT
0144d6a3b7 Do not hold query key in Query. 2021-02-20 22:53:46 +01:00
Camille GILLOT
f96e960ccf Access the session directly from DepContext. 2021-02-20 22:53:46 +01:00
Camille GILLOT
b27266fdb2 Use a QueryContext for try_mark_green. 2021-02-19 17:51:56 +01:00
Camille GILLOT
ea3d465c95 Move try_load_from_on_disk_cache to the QueryContext. 2021-02-19 17:51:55 +01:00
Camille GILLOT
49c1b07a9e Decouple QueryContext from DepContext. 2021-02-19 17:51:49 +01:00
Camille GILLOT
6f04883023 Remove QueryAccessors::to_dep_node. 2021-02-19 17:51:49 +01:00
Camille GILLOT
211b05aef3 Don't require a QueryContext to access the DepGraph. 2021-02-19 17:51:49 +01:00
Eduard-Mihai Burtescu
6165d1cc72 Print -Ztime-passes (and misc stats/logs) on stderr, not stdout. 2021-02-18 14:13:38 +02:00
Tomasz Miąsko
db36db2e81 Inline try_get_cached 2021-02-16 00:00:00 +00:00
bors
d1206f950f Auto merge of #81855 - cjgillot:ensure-cache, r=oli-obk
Check the result cache before the DepGraph when ensuring queries

Split out of https://github.com/rust-lang/rust/pull/70951

Calling `ensure` on already forced queries is a common operation.
Looking at the results cache first is faster than checking the DepGraph for a green node.
2021-02-15 12:11:59 +00:00
klensy
93c8ebe022 bumped smallvec deps 2021-02-14 18:03:11 +03:00
Camille GILLOT
3fc8ed68e9 Check query cache before calling into the query engine. 2021-02-13 21:14:58 +01:00
Camille GILLOT
280a2866d5 Drop the cache lock earlier. 2021-02-13 21:14:58 +01:00
Camille GILLOT
15b0bc6b83 Separate the query cache from the query state. 2021-02-13 21:14:58 +01:00
Camille GILLOT
9f46259a75 Return a Result for query cache. 2021-02-13 21:14:58 +01:00
Camille GILLOT
8684e9e47d Merge {get,ensure}_query. 2021-02-13 21:14:57 +01:00
bors
097bc6a84f Auto merge of #81892 - jyn514:no-inline, r=cjgillot
[experiment] remove `#[inline]` from rustc_query_system::plumbing

These functions have a ton of generic parameters and are instantiated
over and over again. Hopefully this will reduce binary bloat and speed
up bootstrapping times.

r? `@cjgillot`
2021-02-09 18:37:33 +00:00
Mark Rousskov
f564d7abba Switch query descriptions to just String
In practice we never used the borrowed variant anyway.
2021-02-08 17:20:41 -05:00
Joshua Nelson
4f77a1afc2 [experiment] remove #[inline] from rustc_query_system::plumbing
These functions have a ton of generic parameters and are instantiated
over and over again. Hopefully this will reduce binary bloat and speed
up bootstrapping times.
2021-02-08 14:57:15 -05:00
bors
a8f7075532 Auto merge of #80692 - Aaron1011:feature/query-result-debug, r=estebank
Enforce that query results implement Debug

Currently, we require that query keys implement `Debug`, but we do not do the same for query values. This can make incremental compilation bugs difficult to debug - there isn't a good place to print out the result loaded from disk.

This PR adds `Debug` bounds to several query-related functions, allowing us to debug-print the query value when an 'unstable fingerprint' error occurs. This required adding `#[derive(Debug)]` to a fairly large number of types - hopefully, this doesn't have much of an impact on compiler bootstrapping times.
2021-01-26 05:47:23 +00:00
bors
c5a96fb797 Auto merge of #80957 - tgnottingham:direct_serialize_depgraph, r=michaelwoerister
Serialize dependency graph directly from DepGraph

Reduce memory usage by serializing dep graph directly from `DepGraph`,
rather than copying it into `SerializedDepGraph` and serializing that.
2021-01-19 19:36:41 +00:00
Aaron Hill
056fbbf7ee
Undo assertion change 2021-01-16 21:37:53 -05:00
Aaron Hill
6417760632
Run fmt 2021-01-16 17:53:02 -05:00
Aaron Hill
93ab705655
Print result on unstable fingerprint error 2021-01-16 17:53:02 -05:00
Aaron Hill
7afb32557d
Enforce that query results implement Debug 2021-01-16 17:53:02 -05:00
LingMan
a56bffb4f9 Use Option::map_or instead of .map(..).unwrap_or(..) 2021-01-14 19:23:59 +01:00
Tyson Nottingham
09067db8a0 Serialize dependency graph directly from DepGraph
Reduce memory usage by serializing dep graph directly from `DepGraph`,
rather than copying it into `SerializedDepGraph` and serializing that.
2021-01-12 22:20:29 -08:00
Joshua Nelson
0215b3a456 Don't mark force_query_with_job as inline(always)
It's rather large, and using `inline(always)` forces it to be recompiled
in each calling crate.
2021-01-08 18:38:33 -05:00
Camille GILLOT
016ea6b319 Use a side-table of consts instead of matching on the DepKind enum. 2021-01-08 17:48:02 +01:00
Camille GILLOT
d1220fdedf Simplify DepNodeParams. 2021-01-08 17:29:49 +01:00
Tyson Nottingham
03eb75f759 rustc_query_system: avoid race condition when using edge_count 2020-12-22 14:12:57 -08:00
Tyson Nottingham
22ed75158b rustc_query_system: add more comments for dependency graph 2020-12-22 14:12:57 -08:00
Tyson Nottingham
d6b2aaed7d rustc_query_system: rename intern_node to intern_new_node 2020-12-22 14:12:57 -08:00
Tyson Nottingham
712fcae13a rustc_query_system: remove inline annotation from edge_count
This isn't called frequently enough to justify inlining.
2020-12-22 14:12:57 -08:00
Tyson Nottingham
4f76266295 rustc_query_system: minor cleanup
Remove effectively unused parameter and delete out of date comment.
2020-12-22 14:12:57 -08:00
Tyson Nottingham
dd1ab840d2 rustc_query_system: use more space-efficient edges representation
Use single vector of edges rather than per-node vector. There is a small
hit to instruction counts (< 0.5%), but the memory savings make up for it.
2020-12-22 14:12:57 -08:00
Tyson Nottingham
ea47269f5f rustc_query_system: share previous graph edges with current graph
Reduce memory consumption by sharing the previous dependency graph's
edges with the current graph when it is known to be valid to do so. It
is known to be valid whenever we mark a node green because all of its
dependencies were green. It is *not* known to be valid when we mark a
node green because we re-executed its query and its result was the same
as in the previous compilation session. In that case, the dependency set
might have changed (we don't try to determine whether or not it changed
and whether or not we can share).
2020-12-22 14:12:57 -08:00
Tyson Nottingham
f6d6b0c96d rustc_query_system: share previous graph data with current graph
Reduce memory consumption by taking advantage of red/green algorithm
properties to share the previous dependency graph's node data with the
current graph instead of storing node data redundantly. Red nodes can
share the `DepNode`, and green nodes can share the `DepNode` and
`Fingerprint`. Edges will be shared when possible in a later change.
2020-12-22 14:12:57 -08:00
Tyson Nottingham
7795801902 rustc_query_system: explicitly register reused dep nodes
Register nodes that we've reused from the previous session explicitly
with `OnDiskCache`. Previously, we relied on this happening as a side
effect of accessing the nodes in the `PreviousDepGraph`. For the sake of
performance and avoiding unintended side effects, register explictily.
2020-12-18 18:53:12 -08:00
Aaron Hill
3918b82993
Use def_path_hash_to_def_id when re-using a RawDefId
Fixes #79890

Previously, we just copied a `RawDefId` from the 'old' map to the 'new'
map. However, the `RawDefId` for a given `DefPathHash` may be different
in the current compilation session. Using `def_path_hash_to_def_id`
ensures that the `RawDefId` we use is valid in the current session.
2020-12-10 16:04:19 -05:00
Aaron Hill
c2946402ff
Properly re-use def path hash in incremental mode
Fixes #79661

In incremental compilation mode, we update a `DefPathHash -> DefId`
mapping every time we create a `DepNode` for a foreign `DefId`.
This mapping is written out to the on-disk incremental cache, and is
read by the next compilation session to allow us to lazily decode
`DefId`s.

When we decode a `DepNode` from the current incremental cache, we need
to ensure that any previously-recorded `DefPathHash -> DefId` mapping
gets recorded in the new mapping that we write out. However, PR #74967
didn't do this in all cases, leading to us being unable to decode a
`DefPathHash` in certain circumstances.

This PR refactors some of the code around `DepNode` deserialization to
prevent this kind of mistake from happening again.
2020-12-04 22:16:40 -05:00
Aaron Hill
7a9aa4f980
Fix rebase fallout 2020-11-25 15:08:51 -05:00
Aaron Hill
e935d3832c
Lazy DefPath decoding for incremental compilation 2020-11-25 14:49:15 -05:00
Dániel Buga
db8b86b2df Fix typos 2020-11-21 09:06:45 +01:00
Tyson Nottingham
05dde137ca Make PackedFingerprint's Fingerprint private 2020-11-18 15:10:43 -08:00
Tyson Nottingham
f09d474836 Use PackedFingerprint in DepNode to reduce memory consumption 2020-11-18 12:49:09 -08:00
Joshua Nelson
57c6ed0c07 Fix even more clippy warnings 2020-10-30 10:13:39 -04:00
Camille GILLOT
0a4d948b4a Remove unused ProfileCategory. 2020-10-22 22:35:32 +02:00
bors
500ddc5efd Auto merge of #77871 - Julian-Wollersberger:less-query-context, r=oli-obk
Make fewer types generic over QueryContext

While trying to refactor `rustc_query_system::query::QueryContext` to make it dyn-safe, I noticed some smaller things:
* QueryConfig doesn't need to be generic over QueryContext
* ~~The `kind` field on QueryJobId is unused~~
* Some unnecessary where clauses
* Many types in `job.rs` where generic over `QueryContext` but only needed `QueryContext::Query`.
  If handle_cycle_error() could be refactored to not take `error: CycleError<CTX::Query>`, all those bounds could be removed as well.

Changing `find_cycle_in_stack()` in job.rs to not take a `tcx` argument is the only functional change here. Everything else is just updating type signatures. (aka compile-error driven development ^^)

~~Currently there is a weird bug where memory usage suddenly skyrockets when running UI tests. I'll investigate that tomorrow.
A perf run probably won't make sense before that is fixed.~~

EDIT: `kind` actually is used by `Eq`, and re-adding it fixed the memory issue.
2020-10-22 12:24:55 +00:00
Julian Wollersberger
52cedcab92 Remove <CTX: QueryContext> in a bunch of places.
It was only needed by `find_cycle_in_stack()` in job.rs, but needed to be forwarded through dozens of types.
2020-10-19 11:11:09 +02:00
est31
338fad2162 Remove unused code from rustc_query_system 2020-10-14 04:14:32 +02:00
Julian Wollersberger
39b0e79285 Remove generic argument from QueryConfig. 2020-10-12 16:04:49 +02:00
Andreas Jonson
b8752fff19 update the version of itertools and parking_lot
this is to avoid compiling multiple version of the crates in rustc
2020-09-12 08:26:53 +02:00
mark
9e5f7d5631 mv compiler to compiler/ 2020-08-30 18:45:07 +03:00