rust/src/librustc/metadata/cstore.rs

233 lines
7.9 KiB
Rust
Raw Normal View History

// Copyright 2012-2014 The Rust Project Developers. See the COPYRIGHT
// file at the top-level directory of this distribution and at
// http://rust-lang.org/COPYRIGHT.
//
// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
// option. This file may not be copied, modified, or distributed
// except according to those terms.
#[allow(non_camel_case_types)];
2011-07-07 20:39:44 -05:00
// The crate store - a central repo for information collected about external
// crates and libraries
use back::svh::Svh;
use metadata::decoder;
rustc: Optimize reading metadata by 4x We were previously reading metadata via `ar p`, but as learned from rustdoc awhile back, spawning a process to do something is pretty slow. Turns out LLVM has an Archive class to read archives, but it cannot write archives. This commits adds bindings to the read-only version of the LLVM archive class (with a new type that only has a read() method), and then it uses this class when reading the metadata out of rlibs. When you put this in tandem of not compressing the metadata, reading the metadata is 4x faster than it used to be The timings I got for reading metadata from the respective libraries was: libstd-04ff901e-0.9-pre.dylib => 100ms libstd-04ff901e-0.9-pre.rlib => 23ms librustuv-7945354c-0.9-pre.dylib => 4ms librustuv-7945354c-0.9-pre.rlib => 1ms librustc-5b94a16f-0.9-pre.dylib => 87ms librustc-5b94a16f-0.9-pre.rlib => 35ms libextra-a6ebb16f-0.9-pre.dylib => 63ms libextra-a6ebb16f-0.9-pre.rlib => 15ms libsyntax-2e4c0458-0.9-pre.dylib => 86ms libsyntax-2e4c0458-0.9-pre.rlib => 22ms In order to always take advantage of these faster metadata read-times, I sort the files in filesearch based on whether they have an rlib extension or not (prefer all rlib files first). Overall, this halved the compile time for a `fn main() {}` crate from 0.185s to 0.095s on my system (when preferring dynamic linking). Reading metadata is still the slowest pass of the compiler at 0.035s, but it's getting pretty close to linking at 0.021s! The next best optimization is to just not copy the metadata from LLVM because that's the most expensive part of reading metadata right now.
2013-12-16 22:58:21 -06:00
use metadata::loader;
2013-12-20 21:43:27 -06:00
use std::cell::RefCell;
use std::c_vec::CVec;
use collections::HashMap;
2013-03-26 15:38:07 -05:00
use syntax::ast;
use syntax::parse::token::IdentInterner;
use syntax::crateid::CrateId;
// A map from external crate numbers (as decoded from some crate file) to
// local crate numbers (as generated during this session). Each external
// crate may refer to types in other external crates, and each has their
// own crate numbers.
2013-12-21 20:17:29 -06:00
pub type cnum_map = @RefCell<HashMap<ast::CrateNum, ast::CrateNum>>;
pub enum MetadataBlob {
MetadataVec(CVec<u8>),
rustc: Optimize reading metadata by 4x We were previously reading metadata via `ar p`, but as learned from rustdoc awhile back, spawning a process to do something is pretty slow. Turns out LLVM has an Archive class to read archives, but it cannot write archives. This commits adds bindings to the read-only version of the LLVM archive class (with a new type that only has a read() method), and then it uses this class when reading the metadata out of rlibs. When you put this in tandem of not compressing the metadata, reading the metadata is 4x faster than it used to be The timings I got for reading metadata from the respective libraries was: libstd-04ff901e-0.9-pre.dylib => 100ms libstd-04ff901e-0.9-pre.rlib => 23ms librustuv-7945354c-0.9-pre.dylib => 4ms librustuv-7945354c-0.9-pre.rlib => 1ms librustc-5b94a16f-0.9-pre.dylib => 87ms librustc-5b94a16f-0.9-pre.rlib => 35ms libextra-a6ebb16f-0.9-pre.dylib => 63ms libextra-a6ebb16f-0.9-pre.rlib => 15ms libsyntax-2e4c0458-0.9-pre.dylib => 86ms libsyntax-2e4c0458-0.9-pre.rlib => 22ms In order to always take advantage of these faster metadata read-times, I sort the files in filesearch based on whether they have an rlib extension or not (prefer all rlib files first). Overall, this halved the compile time for a `fn main() {}` crate from 0.185s to 0.095s on my system (when preferring dynamic linking). Reading metadata is still the slowest pass of the compiler at 0.035s, but it's getting pretty close to linking at 0.021s! The next best optimization is to just not copy the metadata from LLVM because that's the most expensive part of reading metadata right now.
2013-12-16 22:58:21 -06:00
MetadataArchive(loader::ArchiveMetadata),
}
pub struct crate_metadata {
name: ~str,
data: MetadataBlob,
cnum_map: cnum_map,
cnum: ast::CrateNum
}
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
#[deriving(Eq)]
pub enum LinkagePreference {
RequireDynamic,
RequireStatic,
}
#[deriving(Eq, FromPrimitive)]
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
pub enum NativeLibaryKind {
NativeStatic, // native static library (.a archive)
NativeFramework, // OSX-specific
NativeUnknown, // default way to specify a dynamic library
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
}
// Where a crate came from on the local filesystem. One of these two options
// must be non-None.
2013-12-25 12:10:33 -06:00
#[deriving(Eq, Clone)]
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
pub struct CrateSource {
dylib: Option<Path>,
rlib: Option<Path>,
cnum: ast::CrateNum,
}
pub struct CStore {
2013-12-20 21:43:27 -06:00
priv metas: RefCell<HashMap<ast::CrateNum, @crate_metadata>>,
priv extern_mod_crate_map: RefCell<extern_mod_crate_map>,
priv used_crate_sources: RefCell<Vec<CrateSource> >,
priv used_libraries: RefCell<Vec<(~str, NativeLibaryKind)> >,
priv used_link_args: RefCell<Vec<~str> >,
intr: @IdentInterner
}
// Map from NodeId's of local extern crate statements to crate numbers
type extern_mod_crate_map = HashMap<ast::NodeId, ast::CrateNum>;
2011-07-07 23:37:56 -05:00
2013-12-25 14:08:04 -06:00
impl CStore {
pub fn new(intr: @IdentInterner) -> CStore {
2013-12-25 14:08:04 -06:00
CStore {
2013-12-20 21:43:27 -06:00
metas: RefCell::new(HashMap::new()),
extern_mod_crate_map: RefCell::new(HashMap::new()),
used_crate_sources: RefCell::new(Vec::new()),
used_libraries: RefCell::new(Vec::new()),
used_link_args: RefCell::new(Vec::new()),
2013-12-25 14:08:04 -06:00
intr: intr
}
}
2013-12-25 14:08:04 -06:00
pub fn get_crate_data(&self, cnum: ast::CrateNum) -> @crate_metadata {
2013-12-20 21:43:27 -06:00
let metas = self.metas.borrow();
*metas.get().get(&cnum)
2013-12-25 14:08:04 -06:00
}
pub fn get_crate_hash(&self, cnum: ast::CrateNum) -> Svh {
2013-12-25 14:08:04 -06:00
let cdata = self.get_crate_data(cnum);
decoder::get_crate_hash(cdata.data())
}
pub fn get_crate_id(&self, cnum: ast::CrateNum) -> CrateId {
2013-12-25 14:08:04 -06:00
let cdata = self.get_crate_data(cnum);
decoder::get_crate_id(cdata.data())
2013-12-25 14:08:04 -06:00
}
2013-12-20 22:00:58 -06:00
pub fn set_crate_data(&self, cnum: ast::CrateNum, data: @crate_metadata) {
2013-12-20 21:43:27 -06:00
let mut metas = self.metas.borrow_mut();
metas.get().insert(cnum, data);
2013-12-25 14:08:04 -06:00
}
2013-12-25 14:08:04 -06:00
pub fn have_crate_data(&self, cnum: ast::CrateNum) -> bool {
2013-12-20 21:43:27 -06:00
let metas = self.metas.borrow();
metas.get().contains_key(&cnum)
2013-12-25 14:08:04 -06:00
}
2013-12-25 14:08:04 -06:00
pub fn iter_crate_data(&self, i: |ast::CrateNum, @crate_metadata|) {
2013-12-20 21:43:27 -06:00
let metas = self.metas.borrow();
for (&k, &v) in metas.get().iter() {
2013-12-25 14:08:04 -06:00
i(k, v);
}
}
2013-12-20 22:00:58 -06:00
pub fn add_used_crate_source(&self, src: CrateSource) {
let mut used_crate_sources = self.used_crate_sources.borrow_mut();
if !used_crate_sources.get().contains(&src) {
used_crate_sources.get().push(src);
2013-12-25 14:08:04 -06:00
}
}
2013-12-25 12:10:33 -06:00
pub fn get_used_crate_source(&self, cnum: ast::CrateNum)
-> Option<CrateSource> {
let mut used_crate_sources = self.used_crate_sources.borrow_mut();
used_crate_sources.get().iter().find(|source| source.cnum == cnum)
.map(|source| source.clone())
}
pub fn reset(&self) {
self.metas.with_mut(|s| s.clear());
self.extern_mod_crate_map.with_mut(|s| s.clear());
self.used_crate_sources.with_mut(|s| s.clear());
self.used_libraries.with_mut(|s| s.clear());
self.used_link_args.with_mut(|s| s.clear());
}
// This method is used when generating the command line to pass through to
// system linker. The linker expects undefined symbols on the left of the
// command line to be defined in libraries on the right, not the other way
// around. For more info, see some comments in the add_used_library function
// below.
//
// In order to get this left-to-right dependency ordering, we perform a
// topological sort of all crates putting the leaves at the right-most
// positions.
2013-12-25 14:08:04 -06:00
pub fn get_used_crates(&self, prefer: LinkagePreference)
-> Vec<(ast::CrateNum, Option<Path>)> {
let mut ordering = Vec::new();
fn visit(cstore: &CStore, cnum: ast::CrateNum,
ordering: &mut Vec<ast::CrateNum>) {
if ordering.as_slice().contains(&cnum) { return }
let meta = cstore.get_crate_data(cnum);
for (_, &dep) in meta.cnum_map.borrow().get().iter() {
visit(cstore, dep, ordering);
}
ordering.push(cnum);
};
for (&num, _) in self.metas.borrow().get().iter() {
visit(self, num, &mut ordering);
}
ordering.as_mut_slice().reverse();
let ordering = ordering.as_slice();
let used_crate_sources = self.used_crate_sources.borrow();
let mut libs = used_crate_sources.get()
.iter()
2013-12-25 14:08:04 -06:00
.map(|src| (src.cnum, match prefer {
RequireDynamic => src.dylib.clone(),
RequireStatic => src.rlib.clone(),
}))
.collect::<Vec<(ast::CrateNum, Option<Path>)>>();
libs.sort_by(|&(a, _), &(b, _)| {
ordering.position_elem(&a).cmp(&ordering.position_elem(&b))
});
libs
2013-12-25 14:08:04 -06:00
}
pub fn add_used_library(&self, lib: ~str, kind: NativeLibaryKind) {
2013-12-25 14:08:04 -06:00
assert!(!lib.is_empty());
let mut used_libraries = self.used_libraries.borrow_mut();
used_libraries.get().push((lib, kind));
2013-12-25 14:08:04 -06:00
}
pub fn get_used_libraries<'a>(&'a self)
-> &'a RefCell<Vec<(~str, NativeLibaryKind)> > {
&self.used_libraries
2013-12-25 14:08:04 -06:00
}
2013-12-20 22:00:58 -06:00
pub fn add_used_link_args(&self, args: &str) {
let mut used_link_args = self.used_link_args.borrow_mut();
2013-12-25 14:08:04 -06:00
for s in args.split(' ') {
used_link_args.get().push(s.to_owned());
2013-12-25 14:08:04 -06:00
}
}
pub fn get_used_link_args<'a>(&'a self) -> &'a RefCell<Vec<~str> > {
&self.used_link_args
2013-12-25 14:08:04 -06:00
}
2013-12-20 22:00:58 -06:00
pub fn add_extern_mod_stmt_cnum(&self,
2013-12-25 14:08:04 -06:00
emod_id: ast::NodeId,
cnum: ast::CrateNum) {
let mut extern_mod_crate_map = self.extern_mod_crate_map.borrow_mut();
extern_mod_crate_map.get().insert(emod_id, cnum);
}
2013-12-25 14:08:04 -06:00
pub fn find_extern_mod_stmt_cnum(&self, emod_id: ast::NodeId)
-> Option<ast::CrateNum> {
let extern_mod_crate_map = self.extern_mod_crate_map.borrow();
extern_mod_crate_map.get().find(&emod_id).map(|x| *x)
2013-12-25 14:08:04 -06:00
}
2013-07-02 14:47:32 -05:00
}
impl crate_metadata {
pub fn data<'a>(&'a self) -> &'a [u8] { self.data.as_slice() }
}
impl MetadataBlob {
pub fn as_slice<'a>(&'a self) -> &'a [u8] {
match *self {
MetadataVec(ref vec) => vec.as_slice(),
rustc: Optimize reading metadata by 4x We were previously reading metadata via `ar p`, but as learned from rustdoc awhile back, spawning a process to do something is pretty slow. Turns out LLVM has an Archive class to read archives, but it cannot write archives. This commits adds bindings to the read-only version of the LLVM archive class (with a new type that only has a read() method), and then it uses this class when reading the metadata out of rlibs. When you put this in tandem of not compressing the metadata, reading the metadata is 4x faster than it used to be The timings I got for reading metadata from the respective libraries was: libstd-04ff901e-0.9-pre.dylib => 100ms libstd-04ff901e-0.9-pre.rlib => 23ms librustuv-7945354c-0.9-pre.dylib => 4ms librustuv-7945354c-0.9-pre.rlib => 1ms librustc-5b94a16f-0.9-pre.dylib => 87ms librustc-5b94a16f-0.9-pre.rlib => 35ms libextra-a6ebb16f-0.9-pre.dylib => 63ms libextra-a6ebb16f-0.9-pre.rlib => 15ms libsyntax-2e4c0458-0.9-pre.dylib => 86ms libsyntax-2e4c0458-0.9-pre.rlib => 22ms In order to always take advantage of these faster metadata read-times, I sort the files in filesearch based on whether they have an rlib extension or not (prefer all rlib files first). Overall, this halved the compile time for a `fn main() {}` crate from 0.185s to 0.095s on my system (when preferring dynamic linking). Reading metadata is still the slowest pass of the compiler at 0.035s, but it's getting pretty close to linking at 0.021s! The next best optimization is to just not copy the metadata from LLVM because that's the most expensive part of reading metadata right now.
2013-12-16 22:58:21 -06:00
MetadataArchive(ref ar) => ar.as_slice(),
}
}
}