mikros/rust
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

rustc: Convert dependency_formats to a query

Browse Source 
This commit converts a field of `Session`, `dependency_formats`, into a
query of `TyCtxt`. This information then also needed to be threaded
through to other remaining portions of the linker, but it's relatively
straightforward. The only change here is that instead of
`HashMap<CrateType, T>` the data structure changed to `Vec<(CrateType,
T)>` to make it easier to deal with in queries.
This commit is contained in:
Alex Crichton 2019-09-16 13:34:57 -07:00
parent 66bf391c3a
commit 5d531aeaf4
14 changed files with 455 additions and 419 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

378
src/librustc/middle/dependency_format.rs
View File

@ -1,64 +1,10 @@
//! Resolution of mixing rlibs and dylibs //! Type definitions for learning about the dependency formats of all upstream
//! crates (rlibs/dylibs/oh my).
//! //!
//! When producing a final artifact, such as a dynamic library, the compiler has //! For all the gory details, see the provider of the `dependency_formats`
//! a choice between linking an rlib or linking a dylib of all upstream //! query.
//! dependencies. The linking phase must guarantee, however, that a library only
//! show up once in the object file. For example, it is illegal for library A to
//! be statically linked to B and C in separate dylibs, and then link B and C
//! into a crate D (because library A appears twice).
//!
//! The job of this module is to calculate what format each upstream crate
//! should be used when linking each output type requested in this session. This
//! generally follows this set of rules:
//!
//! 1. Each library must appear exactly once in the output.
//! 2. Each rlib contains only one library (it's just an object file)
//! 3. Each dylib can contain more than one library (due to static linking),
//! and can also bring in many dynamic dependencies.
//!
//! With these constraints in mind, it's generally a very difficult problem to
//! find a solution that's not "all rlibs" or "all dylibs". I have suspicions
//! that NP-ness may come into the picture here...
//!
//! The current selection algorithm below looks mostly similar to:
//!
//! 1. If static linking is required, then require all upstream dependencies
//! to be available as rlibs. If not, generate an error.
//! 2. If static linking is requested (generating an executable), then
//! attempt to use all upstream dependencies as rlibs. If any are not
//! found, bail out and continue to step 3.
//! 3. Static linking has failed, at least one library must be dynamically
//! linked. Apply a heuristic by greedily maximizing the number of
//! dynamically linked libraries.
//! 4. Each upstream dependency available as a dynamic library is
//! registered. The dependencies all propagate, adding to a map. It is
//! possible for a dylib to add a static library as a dependency, but it
//! is illegal for two dylibs to add the same static library as a
//! dependency. The same dylib can be added twice. Additionally, it is
//! illegal to add a static dependency when it was previously found as a
//! dylib (and vice versa)
//! 5. After all dynamic dependencies have been traversed, re-traverse the
//! remaining dependencies and add them statically (if they haven't been
//! added already).
//!
//! While not perfect, this algorithm should help support use-cases such as leaf
//! dependencies being static while the larger tree of inner dependencies are
//! all dynamic. This isn't currently very well battle tested, so it will likely
//! fall short in some use cases.
//!
//! Currently, there is no way to specify the preference of linkage with a
//! particular library (other than a global dynamic/static switch).
//! Additionally, the algorithm is geared towards finding *any* solution rather
//! than finding a number of solutions (there are normally quite a few).
use crate::hir::def_id::CrateNum;
use crate::session::config; use crate::session::config;
use crate::ty::TyCtxt;
use crate::middle::cstore::{self, DepKind};
use crate::middle::cstore::LinkagePreference::{self, RequireStatic, RequireDynamic};
use crate::util::nodemap::FxHashMap;
use rustc_target::spec::PanicStrategy;
/// A list of dependencies for a certain crate type. /// A list of dependencies for a certain crate type.
/// ///
@ -71,324 +17,12 @@ pub type DependencyList = Vec<Linkage>;
/// A mapping of all required dependencies for a particular flavor of output. /// A mapping of all required dependencies for a particular flavor of output.
/// ///
/// This is local to the tcx, and is generally relevant to one session. /// This is local to the tcx, and is generally relevant to one session.
pub type Dependencies = FxHashMap<config::CrateType, DependencyList>; pub type Dependencies = Vec<(config::CrateType, DependencyList)>;
#[derive(Copy, Clone, PartialEq, Debug)] #[derive(Copy, Clone, PartialEq, Debug, HashStable)]
pub enum Linkage { pub enum Linkage {
NotLinked, NotLinked,
IncludedFromDylib, IncludedFromDylib,
Static, Static,
Dynamic, Dynamic,
} }
pub fn calculate(tcx: TyCtxt<'_>) {
let sess = &tcx.sess;
let fmts = sess.crate_types.borrow().iter().map(|&ty| {
let linkage = calculate_type(tcx, ty);
verify_ok(tcx, &linkage);
(ty, linkage)
}).collect::<FxHashMap<_, _>>();
sess.abort_if_errors();
sess.dependency_formats.set(fmts);
}
fn calculate_type(tcx: TyCtxt<'_>, ty: config::CrateType) -> DependencyList {
let sess = &tcx.sess;
if !sess.opts.output_types.should_codegen() {
return Vec::new();
}
let preferred_linkage = match ty {
// cdylibs must have all static dependencies.
config::CrateType::Cdylib => Linkage::Static,
// Generating a dylib without `-C prefer-dynamic` means that we're going
// to try to eagerly statically link all dependencies. This is normally
// done for end-product dylibs, not intermediate products.
config::CrateType::Dylib if !sess.opts.cg.prefer_dynamic => Linkage::Static,
config::CrateType::Dylib => Linkage::Dynamic,
// If the global prefer_dynamic switch is turned off, or the final
// executable will be statically linked, prefer static crate linkage.
config::CrateType::Executable if !sess.opts.cg.prefer_dynamic ||
sess.crt_static() => Linkage::Static,
config::CrateType::Executable => Linkage::Dynamic,
// proc-macro crates are mostly cdylibs, but we also need metadata.
config::CrateType::ProcMacro => Linkage::Static,
// No linkage happens with rlibs, we just needed the metadata (which we
// got long ago), so don't bother with anything.
config::CrateType::Rlib => Linkage::NotLinked,
// staticlibs must have all static dependencies.
config::CrateType::Staticlib => Linkage::Static,
};
if preferred_linkage == Linkage::NotLinked {
// If the crate is not linked, there are no link-time dependencies.
return Vec::new();
}
if preferred_linkage == Linkage::Static {
// Attempt static linkage first. For dylibs and executables, we may be
// able to retry below with dynamic linkage.
if let Some(v) = attempt_static(tcx) {
return v;
}
// Staticlibs, cdylibs, and static executables must have all static
// dependencies. If any are not found, generate some nice pretty errors.
if ty == config::CrateType::Cdylib || ty == config::CrateType::Staticlib ||
(ty == config::CrateType::Executable && sess.crt_static() &&
!sess.target.target.options.crt_static_allows_dylibs) {
for &cnum in tcx.crates().iter() {
if tcx.dep_kind(cnum).macros_only() { continue }
let src = tcx.used_crate_source(cnum);
if src.rlib.is_some() { continue }
sess.err(&format!("crate `{}` required to be available in rlib format, \
but was not found in this form",
tcx.crate_name(cnum)));
}
return Vec::new();
}
}
let mut formats = FxHashMap::default();
// Sweep all crates for found dylibs. Add all dylibs, as well as their
// dependencies, ensuring there are no conflicts. The only valid case for a
// dependency to be relied upon twice is for both cases to rely on a dylib.
for &cnum in tcx.crates().iter() {
if tcx.dep_kind(cnum).macros_only() { continue }
let name = tcx.crate_name(cnum);
let src = tcx.used_crate_source(cnum);
if src.dylib.is_some() {
info!("adding dylib: {}", name);
add_library(tcx, cnum, RequireDynamic, &mut formats);
let deps = tcx.dylib_dependency_formats(cnum);
for &(depnum, style) in deps.iter() {
info!("adding {:?}: {}", style, tcx.crate_name(depnum));
add_library(tcx, depnum, style, &mut formats);
}
}
}
// Collect what we've got so far in the return vector.
let last_crate = tcx.crates().len();
let mut ret = (1..last_crate+1).map(|cnum| {
match formats.get(&CrateNum::new(cnum)) {
Some(&RequireDynamic) => Linkage::Dynamic,
Some(&RequireStatic) => Linkage::IncludedFromDylib,
None => Linkage::NotLinked,
}
}).collect::<Vec<_>>();
// Run through the dependency list again, and add any missing libraries as
// static libraries.
//
// If the crate hasn't been included yet and it's not actually required
// (e.g., it's an allocator) then we skip it here as well.
for &cnum in tcx.crates().iter() {
let src = tcx.used_crate_source(cnum);
if src.dylib.is_none() &&
!formats.contains_key(&cnum) &&
tcx.dep_kind(cnum) == DepKind::Explicit {
assert!(src.rlib.is_some() || src.rmeta.is_some());
info!("adding staticlib: {}", tcx.crate_name(cnum));
add_library(tcx, cnum, RequireStatic, &mut formats);
ret[cnum.as_usize() - 1] = Linkage::Static;
}
}
// We've gotten this far because we're emitting some form of a final
// artifact which means that we may need to inject dependencies of some
// form.
//
// Things like allocators and panic runtimes may not have been activated
// quite yet, so do so here.
activate_injected_dep(*sess.injected_panic_runtime.get(), &mut ret,
&|cnum| tcx.is_panic_runtime(cnum));
// When dylib B links to dylib A, then when using B we must also link to A.
// It could be the case, however, that the rlib for A is present (hence we
// found metadata), but the dylib for A has since been removed.
//
// For situations like this, we perform one last pass over the dependencies,
// making sure that everything is available in the requested format.
for (cnum, kind) in ret.iter().enumerate() {
let cnum = CrateNum::new(cnum + 1);
let src = tcx.used_crate_source(cnum);
match *kind {
Linkage::NotLinked |
Linkage::IncludedFromDylib => {}
Linkage::Static if src.rlib.is_some() => continue,
Linkage::Dynamic if src.dylib.is_some() => continue,
kind => {
let kind = match kind {
Linkage::Static => "rlib",
_ => "dylib",
};
sess.err(&format!("crate `{}` required to be available in {} format, \
but was not found in this form",
tcx.crate_name(cnum), kind));
}
}
}
ret
}
fn add_library(
tcx: TyCtxt<'_>,
cnum: CrateNum,
link: LinkagePreference,
m: &mut FxHashMap<CrateNum, LinkagePreference>,
) {
match m.get(&cnum) {
Some(&link2) => {
// If the linkages differ, then we'd have two copies of the library
// if we continued linking. If the linkages are both static, then we
// would also have two copies of the library (static from two
// different locations).
//
// This error is probably a little obscure, but I imagine that it
// can be refined over time.
if link2 != link || link == RequireStatic {
tcx.sess.struct_err(&format!("cannot satisfy dependencies so `{}` only \
shows up once", tcx.crate_name(cnum)))
.help("having upstream crates all available in one format \
will likely make this go away")
.emit();
}
}
None => { m.insert(cnum, link); }
}
}
fn attempt_static(tcx: TyCtxt<'_>) -> Option<DependencyList> {
let sess = &tcx.sess;
let crates = cstore::used_crates(tcx, RequireStatic);
if !crates.iter().by_ref().all(|&(_, ref p)| p.is_some()) {
return None
}
// All crates are available in an rlib format, so we're just going to link
// everything in explicitly so long as it's actually required.
let last_crate = tcx.crates().len();
let mut ret = (1..last_crate+1).map(|cnum| {
if tcx.dep_kind(CrateNum::new(cnum)) == DepKind::Explicit {
Linkage::Static
} else {
Linkage::NotLinked
}
}).collect::<Vec<_>>();
// Our allocator/panic runtime may not have been linked above if it wasn't
// explicitly linked, which is the case for any injected dependency. Handle
// that here and activate them.
activate_injected_dep(*sess.injected_panic_runtime.get(), &mut ret,
&|cnum| tcx.is_panic_runtime(cnum));
Some(ret)
}
// Given a list of how to link upstream dependencies so far, ensure that an
// injected dependency is activated. This will not do anything if one was
// transitively included already (e.g., via a dylib or explicitly so).
//
// If an injected dependency was not found then we're guaranteed the
// metadata::creader module has injected that dependency (not listed as
// a required dependency) in one of the session's field. If this field is not
// set then this compilation doesn't actually need the dependency and we can
// also skip this step entirely.
fn activate_injected_dep(injected: Option<CrateNum>,
list: &mut DependencyList,
replaces_injected: &dyn Fn(CrateNum) -> bool) {
for (i, slot) in list.iter().enumerate() {
let cnum = CrateNum::new(i + 1);
if !replaces_injected(cnum) {
continue
}
if *slot != Linkage::NotLinked {
return
}
}
if let Some(injected) = injected {
let idx = injected.as_usize() - 1;
assert_eq!(list[idx], Linkage::NotLinked);
list[idx] = Linkage::Static;
}
}
// After the linkage for a crate has been determined we need to verify that
// there's only going to be one allocator in the output.
fn verify_ok(tcx: TyCtxt<'_>, list: &[Linkage]) {
let sess = &tcx.sess;
if list.len() == 0 {
return
}
let mut panic_runtime = None;
for (i, linkage) in list.iter().enumerate() {
if let Linkage::NotLinked = *linkage {
continue
}
let cnum = CrateNum::new(i + 1);
if tcx.is_panic_runtime(cnum) {
if let Some((prev, _)) = panic_runtime {
let prev_name = tcx.crate_name(prev);
let cur_name = tcx.crate_name(cnum);
sess.err(&format!("cannot link together two \
panic runtimes: {} and {}",
prev_name, cur_name));
}
panic_runtime = Some((cnum, tcx.panic_strategy(cnum)));
}
}
// If we found a panic runtime, then we know by this point that it's the
// only one, but we perform validation here that all the panic strategy
// compilation modes for the whole DAG are valid.
if let Some((cnum, found_strategy)) = panic_runtime {
let desired_strategy = sess.panic_strategy();
// First up, validate that our selected panic runtime is indeed exactly
// our same strategy.
if found_strategy != desired_strategy {
sess.err(&format!("the linked panic runtime `{}` is \
not compiled with this crate's \
panic strategy `{}`",
tcx.crate_name(cnum),
desired_strategy.desc()));
}
// Next up, verify that all other crates are compatible with this panic
// strategy. If the dep isn't linked, we ignore it, and if our strategy
// is abort then it's compatible with everything. Otherwise all crates'
// panic strategy must match our own.
for (i, linkage) in list.iter().enumerate() {
if let Linkage::NotLinked = *linkage {
continue
}
if desired_strategy == PanicStrategy::Abort {
continue
}
let cnum = CrateNum::new(i + 1);
let found_strategy = tcx.panic_strategy(cnum);
let is_compiler_builtins = tcx.is_compiler_builtins(cnum);
if is_compiler_builtins || desired_strategy == found_strategy {
continue
}
sess.err(&format!("the crate `{}` is compiled with the \
panic strategy `{}` which is \
incompatible with this crate's \
strategy of `{}`",
tcx.crate_name(cnum),
found_strategy.desc(),
desired_strategy.desc()));
}
}
}

4
src/librustc/query/mod.rs
View File

@ -630,6 +630,10 @@ rustc_queries! {
-> &'tcx [(CrateNum, LinkagePreference)] { -> &'tcx [(CrateNum, LinkagePreference)] {
desc { "dylib dependency formats of crate" } desc { "dylib dependency formats of crate" }
} }
query dependency_formats(_: CrateNum) -> Lrc<crate::middle::dependency_format::Dependencies> {
desc { "get the linkage format of all dependencies" }
}
} }
Codegen { Codegen {

2
src/librustc/session/config.rs
View File

@ -687,7 +687,7 @@ pub enum EntryFnType {
impl_stable_hash_via_hash!(EntryFnType); impl_stable_hash_via_hash!(EntryFnType);
#[derive(Copy, PartialEq, PartialOrd, Clone, Ord, Eq, Hash, Debug)] #[derive(Copy, PartialEq, PartialOrd, Clone, Ord, Eq, Hash, Debug, HashStable)]
pub enum CrateType { pub enum CrateType {
Executable, Executable,
Dylib, Dylib,

3
src/librustc/session/mod.rs
View File

@ -7,7 +7,6 @@ use rustc_data_structures::fingerprint::Fingerprint;
use crate::lint; use crate::lint;
use crate::lint::builtin::BuiltinLintDiagnostics; use crate::lint::builtin::BuiltinLintDiagnostics;
use crate::middle::dependency_format;
use crate::session::config::{OutputType, PrintRequest, SwitchWithOptPath}; use crate::session::config::{OutputType, PrintRequest, SwitchWithOptPath};
use crate::session::search_paths::{PathKind, SearchPath}; use crate::session::search_paths::{PathKind, SearchPath};
use crate::util::nodemap::{FxHashMap, FxHashSet}; use crate::util::nodemap::{FxHashMap, FxHashSet};
@ -91,7 +90,6 @@ pub struct Session {
pub plugin_llvm_passes: OneThread<RefCell<Vec<String>>>, pub plugin_llvm_passes: OneThread<RefCell<Vec<String>>>,
pub plugin_attributes: Lock<Vec<(Symbol, AttributeType)>>, pub plugin_attributes: Lock<Vec<(Symbol, AttributeType)>>,
pub crate_types: Once<Vec<config::CrateType>>, pub crate_types: Once<Vec<config::CrateType>>,
pub dependency_formats: Once<dependency_format::Dependencies>,
/// The `crate_disambiguator` is constructed out of all the `-C metadata` /// The `crate_disambiguator` is constructed out of all the `-C metadata`
/// arguments passed to the compiler. Its value together with the crate-name /// arguments passed to the compiler. Its value together with the crate-name
/// forms a unique global identifier for the crate. It is used to allow /// forms a unique global identifier for the crate. It is used to allow
@ -1247,7 +1245,6 @@ fn build_session_(
plugin_llvm_passes: OneThread::new(RefCell::new(Vec::new())), plugin_llvm_passes: OneThread::new(RefCell::new(Vec::new())),
plugin_attributes: Lock::new(Vec::new()), plugin_attributes: Lock::new(Vec::new()),
crate_types: Once::new(), crate_types: Once::new(),
dependency_formats: Once::new(),
crate_disambiguator: Once::new(), crate_disambiguator: Once::new(),
features: Once::new(), features: Once::new(),
recursion_limit: Once::new(), recursion_limit: Once::new(),

61
src/librustc_codegen_ssa/back/link.rs
View File

@ -219,15 +219,24 @@ pub fn get_linker(sess: &Session, linker: &Path, flavor: LinkerFlavor) -> (PathB
(linker.to_path_buf(), cmd) (linker.to_path_buf(), cmd)
} }
pub fn each_linked_rlib(sess: &Session, pub fn each_linked_rlib(
info: &CrateInfo, info: &CrateInfo,
f: &mut dyn FnMut(CrateNum, &Path)) -> Result<(), String> { f: &mut dyn FnMut(CrateNum, &Path),
) -> Result<(), String> {
let crates = info.used_crates_static.iter(); let crates = info.used_crates_static.iter();
let fmts = sess.dependency_formats.borrow(); let mut fmts = None;
let fmts = fmts.get(&config::CrateType::Executable) for (ty, list) in info.dependency_formats.iter() {
.or_else(|| fmts.get(&config::CrateType::Staticlib)) match ty {
.or_else(|| fmts.get(&config::CrateType::Cdylib)) config::CrateType::Executable |
.or_else(|| fmts.get(&config::CrateType::ProcMacro)); config::CrateType::Staticlib |
config::CrateType::Cdylib |
config::CrateType::ProcMacro => {
fmts = Some(list);
break;
}
_ => {}
}
}
let fmts = match fmts { let fmts = match fmts {
Some(f) => f, Some(f) => f,
None => return Err("could not find formats for rlibs".to_string()) None => return Err("could not find formats for rlibs".to_string())
@ -406,7 +415,7 @@ fn link_staticlib<'a, B: ArchiveBuilder<'a>>(sess: &'a Session,
tempdir); tempdir);
let mut all_native_libs = vec![]; let mut all_native_libs = vec![];
let res = each_linked_rlib(sess, &codegen_results.crate_info, &mut |cnum, path| { let res = each_linked_rlib(&codegen_results.crate_info, &mut |cnum, path| {
let name = &codegen_results.crate_info.crate_name[&cnum]; let name = &codegen_results.crate_info.crate_name[&cnum];
let native_libs = &codegen_results.crate_info.native_libraries[&cnum]; let native_libs = &codegen_results.crate_info.native_libraries[&cnum];
@ -1294,11 +1303,13 @@ pub fn add_local_native_libraries(cmd: &mut dyn Linker,
// Rust crates are not considered at all when creating an rlib output. All // Rust crates are not considered at all when creating an rlib output. All
// dependencies will be linked when producing the final output (instead of // dependencies will be linked when producing the final output (instead of
// the intermediate rlib version) // the intermediate rlib version)
fn add_upstream_rust_crates<'a, B: ArchiveBuilder<'a>>(cmd: &mut dyn Linker, fn add_upstream_rust_crates<'a, B: ArchiveBuilder<'a>>(
sess: &'a Session, cmd: &mut dyn Linker,
codegen_results: &CodegenResults, sess: &'a Session,
crate_type: config::CrateType, codegen_results: &CodegenResults,
tmpdir: &Path) { crate_type: config::CrateType,
tmpdir: &Path,
) {
// All of the heavy lifting has previously been accomplished by the // All of the heavy lifting has previously been accomplished by the
// dependency_format module of the compiler. This is just crawling the // dependency_format module of the compiler. This is just crawling the
// output of that module, adding crates as necessary. // output of that module, adding crates as necessary.
@ -1307,8 +1318,10 @@ fn add_upstream_rust_crates<'a, B: ArchiveBuilder<'a>>(cmd: &mut dyn Linker,
// will slurp up the object files inside), and linking to a dynamic library // will slurp up the object files inside), and linking to a dynamic library
// involves just passing the right -l flag. // involves just passing the right -l flag.
let formats = sess.dependency_formats.borrow(); let (_, data) = codegen_results.crate_info.dependency_formats
let data = formats.get(&crate_type).unwrap(); .iter()
.find(|(ty, _)| *ty == crate_type)
.expect("failed to find crate type in dependency format list");
// Invoke get_used_crates to ensure that we get a topological sorting of // Invoke get_used_crates to ensure that we get a topological sorting of
// crates. // crates.
@ -1620,10 +1633,12 @@ fn add_upstream_rust_crates<'a, B: ArchiveBuilder<'a>>(cmd: &mut dyn Linker,
// generic function calls a native function, then the generic function must // generic function calls a native function, then the generic function must
// be instantiated in the target crate, meaning that the native symbol must // be instantiated in the target crate, meaning that the native symbol must
// also be resolved in the target crate. // also be resolved in the target crate.
pub fn add_upstream_native_libraries(cmd: &mut dyn Linker, pub fn add_upstream_native_libraries(
sess: &Session, cmd: &mut dyn Linker,
codegen_results: &CodegenResults, sess: &Session,
crate_type: config::CrateType) { codegen_results: &CodegenResults,
crate_type: config::CrateType,
) {
// Be sure to use a topological sorting of crates because there may be // Be sure to use a topological sorting of crates because there may be
// interdependencies between native libraries. When passing -nodefaultlibs, // interdependencies between native libraries. When passing -nodefaultlibs,
// for example, almost all native libraries depend on libc, so we have to // for example, almost all native libraries depend on libc, so we have to
@ -1633,8 +1648,10 @@ pub fn add_upstream_native_libraries(cmd: &mut dyn Linker,
// This passes RequireStatic, but the actual requirement doesn't matter, // This passes RequireStatic, but the actual requirement doesn't matter,
// we're just getting an ordering of crate numbers, we're not worried about // we're just getting an ordering of crate numbers, we're not worried about
// the paths. // the paths.
let formats = sess.dependency_formats.borrow(); let (_, data) = codegen_results.crate_info.dependency_formats
let data = formats.get(&crate_type).unwrap(); .iter()
.find(|(ty, _)| *ty == crate_type)
.expect("failed to find crate type in dependency format list");
let crates = &codegen_results.crate_info.used_crates_static; let crates = &codegen_results.crate_info.used_crates_static;
for &(cnum, _) in crates { for &(cnum, _) in crates {

12
src/librustc_codegen_ssa/back/linker.rs
View File

@ -1092,10 +1092,16 @@ fn exported_symbols(tcx: TyCtxt<'_>, crate_type: CrateType) -> Vec<String> {
} }
} }
let formats = tcx.sess.dependency_formats.borrow(); let formats = tcx.dependency_formats(LOCAL_CRATE);
let deps = formats[&crate_type].iter(); let deps = formats.iter().filter_map(|(t, list)| {
if *t == crate_type {
Some(list)
} else {
None
}
}).next().unwrap();
for (index, dep_format) in deps.enumerate() { for (index, dep_format) in deps.iter().enumerate() {
let cnum = CrateNum::new(index + 1); let cnum = CrateNum::new(index + 1);
// For each dependency that we are linking to statically ... // For each dependency that we are linking to statically ...
if *dep_format == Linkage::Static { if *dep_format == Linkage::Static {

2
src/librustc_codegen_ssa/back/write.rs
View File

@ -1048,7 +1048,7 @@ fn start_executing_work<B: ExtraBackendMethods>(
}).expect("failed to spawn helper thread"); }).expect("failed to spawn helper thread");
let mut each_linked_rlib_for_lto = Vec::new(); let mut each_linked_rlib_for_lto = Vec::new();
drop(link::each_linked_rlib(sess, crate_info, &mut |cnum, path| { drop(link::each_linked_rlib(crate_info, &mut |cnum, path| {
if link::ignored_for_lto(sess, crate_info, cnum) { if link::ignored_for_lto(sess, crate_info, cnum) {
return return
} }

3
src/librustc_codegen_ssa/base.rs
View File

@ -539,7 +539,7 @@ pub fn codegen_crate<B: ExtraBackendMethods>(
// linkage, then it's already got an allocator shim and we'll be using that // linkage, then it's already got an allocator shim and we'll be using that
// one instead. If nothing exists then it's our job to generate the // one instead. If nothing exists then it's our job to generate the
// allocator! // allocator!
let any_dynamic_crate = tcx.sess.dependency_formats.borrow() let any_dynamic_crate = tcx.dependency_formats(LOCAL_CRATE)
.iter() .iter()
.any(|(_, list)| { .any(|(_, list)| {
use rustc::middle::dependency_format::Linkage; use rustc::middle::dependency_format::Linkage;
@ -731,6 +731,7 @@ impl CrateInfo {
used_crate_source: Default::default(), used_crate_source: Default::default(),
lang_item_to_crate: Default::default(), lang_item_to_crate: Default::default(),
missing_lang_items: Default::default(), missing_lang_items: Default::default(),
dependency_formats: tcx.dependency_formats(LOCAL_CRATE),
}; };
let lang_items = tcx.lang_items(); let lang_items = tcx.lang_items();

2
src/librustc_codegen_ssa/lib.rs
View File

@ -33,6 +33,7 @@ use rustc_data_structures::fx::{FxHashMap, FxHashSet};
use rustc_data_structures::sync::Lrc; use rustc_data_structures::sync::Lrc;
use rustc_data_structures::svh::Svh; use rustc_data_structures::svh::Svh;
use rustc::middle::cstore::{LibSource, CrateSource, NativeLibrary}; use rustc::middle::cstore::{LibSource, CrateSource, NativeLibrary};
use rustc::middle::dependency_format::Dependencies;
use syntax_pos::symbol::Symbol; use syntax_pos::symbol::Symbol;
mod error_codes; mod error_codes;
@ -142,6 +143,7 @@ pub struct CrateInfo {
pub used_crates_dynamic: Vec<(CrateNum, LibSource)>, pub used_crates_dynamic: Vec<(CrateNum, LibSource)>,
pub lang_item_to_crate: FxHashMap<LangItem, CrateNum>, pub lang_item_to_crate: FxHashMap<LangItem, CrateNum>,
pub missing_lang_items: FxHashMap<CrateNum, Vec<LangItem>>, pub missing_lang_items: FxHashMap<CrateNum, Vec<LangItem>>,
pub dependency_formats: Lrc<Dependencies>,
} }

4
src/librustc_interface/passes.rs
View File

@ -1079,10 +1079,6 @@ pub fn start_codegen<'tcx>(
tcx.print_debug_stats(); tcx.print_debug_stats();
} }
time(tcx.sess, "resolving dependency formats", || {
middle::dependency_format::calculate(tcx)
});
let (metadata, need_metadata_module) = time(tcx.sess, "metadata encoding and writing", || { let (metadata, need_metadata_module) = time(tcx.sess, "metadata encoding and writing", || {
encode_and_write_metadata(tcx, outputs) encode_and_write_metadata(tcx, outputs)
}); });

5
src/librustc_metadata/cstore_impl.rs
View File

@ -370,6 +370,11 @@ pub fn provide(providers: &mut Providers<'_>) {
tcx.arena.alloc(visible_parent_map) tcx.arena.alloc(visible_parent_map)
}, },
dependency_formats: |tcx, cnum| {
assert_eq!(cnum, LOCAL_CRATE);
Lrc::new(crate::dependency_format::calculate(tcx))
},
..*providers ..*providers
}; };
} }

371
src/librustc_metadata/dependency_format.rs Normal file
View File

@ -0,0 +1,371 @@
//! Resolution of mixing rlibs and dylibs
//!
//! When producing a final artifact, such as a dynamic library, the compiler has
//! a choice between linking an rlib or linking a dylib of all upstream
//! dependencies. The linking phase must guarantee, however, that a library only
//! show up once in the object file. For example, it is illegal for library A to
//! be statically linked to B and C in separate dylibs, and then link B and C
//! into a crate D (because library A appears twice).
//!
//! The job of this module is to calculate what format each upstream crate
//! should be used when linking each output type requested in this session. This
//! generally follows this set of rules:
//!
//! 1. Each library must appear exactly once in the output.
//! 2. Each rlib contains only one library (it's just an object file)
//! 3. Each dylib can contain more than one library (due to static linking),
//! and can also bring in many dynamic dependencies.
//!
//! With these constraints in mind, it's generally a very difficult problem to
//! find a solution that's not "all rlibs" or "all dylibs". I have suspicions
//! that NP-ness may come into the picture here...
//!
//! The current selection algorithm below looks mostly similar to:
//!
//! 1. If static linking is required, then require all upstream dependencies
//! to be available as rlibs. If not, generate an error.
//! 2. If static linking is requested (generating an executable), then
//! attempt to use all upstream dependencies as rlibs. If any are not
//! found, bail out and continue to step 3.
//! 3. Static linking has failed, at least one library must be dynamically
//! linked. Apply a heuristic by greedily maximizing the number of
//! dynamically linked libraries.
//! 4. Each upstream dependency available as a dynamic library is
//! registered. The dependencies all propagate, adding to a map. It is
//! possible for a dylib to add a static library as a dependency, but it
//! is illegal for two dylibs to add the same static library as a
//! dependency. The same dylib can be added twice. Additionally, it is
//! illegal to add a static dependency when it was previously found as a
//! dylib (and vice versa)
//! 5. After all dynamic dependencies have been traversed, re-traverse the
//! remaining dependencies and add them statically (if they haven't been
//! added already).
//!
//! While not perfect, this algorithm should help support use-cases such as leaf
//! dependencies being static while the larger tree of inner dependencies are
//! all dynamic. This isn't currently very well battle tested, so it will likely
//! fall short in some use cases.
//!
//! Currently, there is no way to specify the preference of linkage with a
//! particular library (other than a global dynamic/static switch).
//! Additionally, the algorithm is geared towards finding *any* solution rather
//! than finding a number of solutions (there are normally quite a few).
use rustc::hir::def_id::CrateNum;
use rustc::middle::cstore::LinkagePreference::{self, RequireStatic, RequireDynamic};
use rustc::middle::cstore::{self, DepKind};
use rustc::middle::dependency_format::{DependencyList, Dependencies, Linkage};
use rustc::session::config;
use rustc::ty::TyCtxt;
use rustc::util::nodemap::FxHashMap;
use rustc_target::spec::PanicStrategy;
pub fn calculate(tcx: TyCtxt<'_>) -> Dependencies {
tcx.sess.crate_types.borrow().iter().map(|&ty| {
let linkage = calculate_type(tcx, ty);
verify_ok(tcx, &linkage);
(ty, linkage)
}).collect::<Vec<_>>()
}
fn calculate_type(tcx: TyCtxt<'_>, ty: config::CrateType) -> DependencyList {
let sess = &tcx.sess;
if !sess.opts.output_types.should_codegen() {
return Vec::new();
}
let preferred_linkage = match ty {
// cdylibs must have all static dependencies.
config::CrateType::Cdylib => Linkage::Static,
// Generating a dylib without `-C prefer-dynamic` means that we're going
// to try to eagerly statically link all dependencies. This is normally
// done for end-product dylibs, not intermediate products.
config::CrateType::Dylib if !sess.opts.cg.prefer_dynamic => Linkage::Static,
config::CrateType::Dylib => Linkage::Dynamic,
// If the global prefer_dynamic switch is turned off, or the final
// executable will be statically linked, prefer static crate linkage.
config::CrateType::Executable if !sess.opts.cg.prefer_dynamic ||
sess.crt_static() => Linkage::Static,
config::CrateType::Executable => Linkage::Dynamic,
// proc-macro crates are mostly cdylibs, but we also need metadata.
config::CrateType::ProcMacro => Linkage::Static,
// No linkage happens with rlibs, we just needed the metadata (which we
// got long ago), so don't bother with anything.
config::CrateType::Rlib => Linkage::NotLinked,
// staticlibs must have all static dependencies.
config::CrateType::Staticlib => Linkage::Static,
};
if preferred_linkage == Linkage::NotLinked {
// If the crate is not linked, there are no link-time dependencies.
return Vec::new();
}
if preferred_linkage == Linkage::Static {
// Attempt static linkage first. For dylibs and executables, we may be
// able to retry below with dynamic linkage.
if let Some(v) = attempt_static(tcx) {
return v;
}
// Staticlibs, cdylibs, and static executables must have all static
// dependencies. If any are not found, generate some nice pretty errors.
if ty == config::CrateType::Cdylib || ty == config::CrateType::Staticlib ||
(ty == config::CrateType::Executable && sess.crt_static() &&
!sess.target.target.options.crt_static_allows_dylibs) {
for &cnum in tcx.crates().iter() {
if tcx.dep_kind(cnum).macros_only() { continue }
let src = tcx.used_crate_source(cnum);
if src.rlib.is_some() { continue }
sess.err(&format!("crate `{}` required to be available in rlib format, \
but was not found in this form",
tcx.crate_name(cnum)));
}
return Vec::new();
}
}
let mut formats = FxHashMap::default();
// Sweep all crates for found dylibs. Add all dylibs, as well as their
// dependencies, ensuring there are no conflicts. The only valid case for a
// dependency to be relied upon twice is for both cases to rely on a dylib.
for &cnum in tcx.crates().iter() {
if tcx.dep_kind(cnum).macros_only() { continue }
let name = tcx.crate_name(cnum);
let src = tcx.used_crate_source(cnum);
if src.dylib.is_some() {
log::info!("adding dylib: {}", name);
add_library(tcx, cnum, RequireDynamic, &mut formats);
let deps = tcx.dylib_dependency_formats(cnum);
for &(depnum, style) in deps.iter() {
log::info!("adding {:?}: {}", style, tcx.crate_name(depnum));
add_library(tcx, depnum, style, &mut formats);
}
}
}
// Collect what we've got so far in the return vector.
let last_crate = tcx.crates().len();
let mut ret = (1..last_crate+1).map(|cnum| {
match formats.get(&CrateNum::new(cnum)) {
Some(&RequireDynamic) => Linkage::Dynamic,
Some(&RequireStatic) => Linkage::IncludedFromDylib,
None => Linkage::NotLinked,
}
}).collect::<Vec<_>>();
// Run through the dependency list again, and add any missing libraries as
// static libraries.
//
// If the crate hasn't been included yet and it's not actually required
// (e.g., it's an allocator) then we skip it here as well.
for &cnum in tcx.crates().iter() {
let src = tcx.used_crate_source(cnum);
if src.dylib.is_none() &&
!formats.contains_key(&cnum) &&
tcx.dep_kind(cnum) == DepKind::Explicit {
assert!(src.rlib.is_some() || src.rmeta.is_some());
log::info!("adding staticlib: {}", tcx.crate_name(cnum));
add_library(tcx, cnum, RequireStatic, &mut formats);
ret[cnum.as_usize() - 1] = Linkage::Static;
}
}
// We've gotten this far because we're emitting some form of a final
// artifact which means that we may need to inject dependencies of some
// form.
//
// Things like allocators and panic runtimes may not have been activated
// quite yet, so do so here.
activate_injected_dep(*sess.injected_panic_runtime.get(), &mut ret,
&|cnum| tcx.is_panic_runtime(cnum));
// When dylib B links to dylib A, then when using B we must also link to A.
// It could be the case, however, that the rlib for A is present (hence we
// found metadata), but the dylib for A has since been removed.
//
// For situations like this, we perform one last pass over the dependencies,
// making sure that everything is available in the requested format.
for (cnum, kind) in ret.iter().enumerate() {
let cnum = CrateNum::new(cnum + 1);
let src = tcx.used_crate_source(cnum);
match *kind {
Linkage::NotLinked |
Linkage::IncludedFromDylib => {}
Linkage::Static if src.rlib.is_some() => continue,
Linkage::Dynamic if src.dylib.is_some() => continue,
kind => {
let kind = match kind {
Linkage::Static => "rlib",
_ => "dylib",
};
sess.err(&format!("crate `{}` required to be available in {} format, \
but was not found in this form",
tcx.crate_name(cnum), kind));
}
}
}
ret
}
fn add_library(
tcx: TyCtxt<'_>,
cnum: CrateNum,
link: LinkagePreference,
m: &mut FxHashMap<CrateNum, LinkagePreference>,
) {
match m.get(&cnum) {
Some(&link2) => {
// If the linkages differ, then we'd have two copies of the library
// if we continued linking. If the linkages are both static, then we
// would also have two copies of the library (static from two
// different locations).
//
// This error is probably a little obscure, but I imagine that it
// can be refined over time.
if link2 != link || link == RequireStatic {
tcx.sess.struct_err(&format!("cannot satisfy dependencies so `{}` only \
shows up once", tcx.crate_name(cnum)))
.help("having upstream crates all available in one format \
will likely make this go away")
.emit();
}
}
None => { m.insert(cnum, link); }
}
}
fn attempt_static(tcx: TyCtxt<'_>) -> Option<DependencyList> {
let sess = &tcx.sess;
let crates = cstore::used_crates(tcx, RequireStatic);
if !crates.iter().by_ref().all(|&(_, ref p)| p.is_some()) {
return None
}
// All crates are available in an rlib format, so we're just going to link
// everything in explicitly so long as it's actually required.
let last_crate = tcx.crates().len();
let mut ret = (1..last_crate+1).map(|cnum| {
if tcx.dep_kind(CrateNum::new(cnum)) == DepKind::Explicit {
Linkage::Static
} else {
Linkage::NotLinked
}
}).collect::<Vec<_>>();
// Our allocator/panic runtime may not have been linked above if it wasn't
// explicitly linked, which is the case for any injected dependency. Handle
// that here and activate them.
activate_injected_dep(*sess.injected_panic_runtime.get(), &mut ret,
&|cnum| tcx.is_panic_runtime(cnum));
Some(ret)
}
// Given a list of how to link upstream dependencies so far, ensure that an
// injected dependency is activated. This will not do anything if one was
// transitively included already (e.g., via a dylib or explicitly so).
//
// If an injected dependency was not found then we're guaranteed the
// metadata::creader module has injected that dependency (not listed as
// a required dependency) in one of the session's field. If this field is not
// set then this compilation doesn't actually need the dependency and we can
// also skip this step entirely.
fn activate_injected_dep(injected: Option<CrateNum>,
list: &mut DependencyList,
replaces_injected: &dyn Fn(CrateNum) -> bool) {
for (i, slot) in list.iter().enumerate() {
let cnum = CrateNum::new(i + 1);
if !replaces_injected(cnum) {
continue
}
if *slot != Linkage::NotLinked {
return
}
}
if let Some(injected) = injected {
let idx = injected.as_usize() - 1;
assert_eq!(list[idx], Linkage::NotLinked);
list[idx] = Linkage::Static;
}
}
// After the linkage for a crate has been determined we need to verify that
// there's only going to be one allocator in the output.
fn verify_ok(tcx: TyCtxt<'_>, list: &[Linkage]) {
let sess = &tcx.sess;
if list.len() == 0 {
return
}
let mut panic_runtime = None;
for (i, linkage) in list.iter().enumerate() {
if let Linkage::NotLinked = *linkage {
continue
}
let cnum = CrateNum::new(i + 1);
if tcx.is_panic_runtime(cnum) {
if let Some((prev, _)) = panic_runtime {
let prev_name = tcx.crate_name(prev);
let cur_name = tcx.crate_name(cnum);
sess.err(&format!("cannot link together two \
panic runtimes: {} and {}",
prev_name, cur_name));
}
panic_runtime = Some((cnum, tcx.panic_strategy(cnum)));
}
}
// If we found a panic runtime, then we know by this point that it's the
// only one, but we perform validation here that all the panic strategy
// compilation modes for the whole DAG are valid.
if let Some((cnum, found_strategy)) = panic_runtime {
let desired_strategy = sess.panic_strategy();
// First up, validate that our selected panic runtime is indeed exactly
// our same strategy.
if found_strategy != desired_strategy {
sess.err(&format!("the linked panic runtime `{}` is \
not compiled with this crate's \
panic strategy `{}`",
tcx.crate_name(cnum),
desired_strategy.desc()));
}
// Next up, verify that all other crates are compatible with this panic
// strategy. If the dep isn't linked, we ignore it, and if our strategy
// is abort then it's compatible with everything. Otherwise all crates'
// panic strategy must match our own.
for (i, linkage) in list.iter().enumerate() {
if let Linkage::NotLinked = *linkage {
continue
}
if desired_strategy == PanicStrategy::Abort {
continue
}
let cnum = CrateNum::new(i + 1);
let found_strategy = tcx.panic_strategy(cnum);
let is_compiler_builtins = tcx.is_compiler_builtins(cnum);
if is_compiler_builtins || desired_strategy == found_strategy {
continue
}
sess.err(&format!("the crate `{}` is compiled with the \
panic strategy `{}` which is \
incompatible with this crate's \
strategy of `{}`",
tcx.crate_name(cnum),
found_strategy.desc(),
desired_strategy.desc()));
}
}
}

26
src/librustc_metadata/encoder.rs
View File

@ -1649,20 +1649,22 @@ impl EncodeContext<'tcx> {
} }
fn encode_dylib_dependency_formats(&mut self) -> Lazy<[Option<LinkagePreference>]> { fn encode_dylib_dependency_formats(&mut self) -> Lazy<[Option<LinkagePreference>]> {
match self.tcx.sess.dependency_formats.borrow().get(&config::CrateType::Dylib) { let formats = self.tcx.dependency_formats(LOCAL_CRATE);
Some(arr) => { for (ty, arr) in formats.iter() {
self.lazy(arr.iter().map(|slot| { if *ty != config::CrateType::Dylib {
match *slot { continue;
Linkage::NotLinked |
Linkage::IncludedFromDylib => None,
Linkage::Dynamic => Some(LinkagePreference::RequireDynamic),
Linkage::Static => Some(LinkagePreference::RequireStatic),
}
}))
} }
None => Lazy::empty(), return self.lazy(arr.iter().map(|slot| {
match *slot {
Linkage::NotLinked |
Linkage::IncludedFromDylib => None,
Linkage::Dynamic => Some(LinkagePreference::RequireDynamic),
Linkage::Static => Some(LinkagePreference::RequireStatic),
}
}));
} }
Lazy::empty()
} }
fn encode_info_for_foreign_item(&mut self, fn encode_info_for_foreign_item(&mut self,

1
src/librustc_metadata/lib.rs
View File

@ -32,6 +32,7 @@ mod schema;
mod native_libs; mod native_libs;
mod link_args; mod link_args;
mod foreign_modules; mod foreign_modules;
mod dependency_format;
pub mod creader; pub mod creader;
pub mod cstore; pub mod cstore;