rust/src/librustc/back/archive.rs

264 lines
9.3 KiB
Rust
Raw Normal View History

// Copyright 2013-2014 The Rust Project Developers. See the COPYRIGHT
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
// file at the top-level directory of this distribution and at
// http://rust-lang.org/COPYRIGHT.
//
// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
// option. This file may not be copied, modified, or distributed
// except according to those terms.
//! A helper class for dealing with static archives
2014-07-04 21:41:54 -05:00
use super::link::{get_ar_prog};
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
use driver::session::Session;
use metadata::filesearch;
2014-07-04 21:41:54 -05:00
use llvm::{ArchiveRef, llvm};
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
2014-02-26 11:58:41 -06:00
use libc;
use std::io::process::{Command, ProcessOutput};
core: Remove the cast module This commit revisits the `cast` module in libcore and libstd, and scrutinizes all functions inside of it. The result was to remove the `cast` module entirely, folding all functionality into the `mem` module. Specifically, this is the fate of each function in the `cast` module. * transmute - This function was moved to `mem`, but it is now marked as #[unstable]. This is due to planned changes to the `transmute` function and how it can be invoked (see the #[unstable] comment). For more information, see RFC 5 and #12898 * transmute_copy - This function was moved to `mem`, with clarification that is is not an error to invoke it with T/U that are different sizes, but rather that it is strongly discouraged. This function is now #[stable] * forget - This function was moved to `mem` and marked #[stable] * bump_box_refcount - This function was removed due to the deprecation of managed boxes as well as its questionable utility. * transmute_mut - This function was previously deprecated, and removed as part of this commit. * transmute_mut_unsafe - This function doesn't serve much of a purpose when it can be achieved with an `as` in safe code, so it was removed. * transmute_lifetime - This function was removed because it is likely a strong indication that code is incorrect in the first place. * transmute_mut_lifetime - This function was removed for the same reasons as `transmute_lifetime` * copy_lifetime - This function was moved to `mem`, but it is marked `#[unstable]` now due to the likelihood of being removed in the future if it is found to not be very useful. * copy_mut_lifetime - This function was also moved to `mem`, but had the same treatment as `copy_lifetime`. * copy_lifetime_vec - This function was removed because it is not used today, and its existence is not necessary with DST (copy_lifetime will suffice). In summary, the cast module was stripped down to these functions, and then the functions were moved to the `mem` module. transmute - #[unstable] transmute_copy - #[stable] forget - #[stable] copy_lifetime - #[unstable] copy_mut_lifetime - #[unstable] [breaking-change]
2014-05-09 12:34:51 -05:00
use std::io::{fs, TempDir};
use std::io;
use std::mem;
use std::os;
2014-02-16 02:04:33 -06:00
use std::raw;
core: Remove the cast module This commit revisits the `cast` module in libcore and libstd, and scrutinizes all functions inside of it. The result was to remove the `cast` module entirely, folding all functionality into the `mem` module. Specifically, this is the fate of each function in the `cast` module. * transmute - This function was moved to `mem`, but it is now marked as #[unstable]. This is due to planned changes to the `transmute` function and how it can be invoked (see the #[unstable] comment). For more information, see RFC 5 and #12898 * transmute_copy - This function was moved to `mem`, with clarification that is is not an error to invoke it with T/U that are different sizes, but rather that it is strongly discouraged. This function is now #[stable] * forget - This function was moved to `mem` and marked #[stable] * bump_box_refcount - This function was removed due to the deprecation of managed boxes as well as its questionable utility. * transmute_mut - This function was previously deprecated, and removed as part of this commit. * transmute_mut_unsafe - This function doesn't serve much of a purpose when it can be achieved with an `as` in safe code, so it was removed. * transmute_lifetime - This function was removed because it is likely a strong indication that code is incorrect in the first place. * transmute_mut_lifetime - This function was removed for the same reasons as `transmute_lifetime` * copy_lifetime - This function was moved to `mem`, but it is marked `#[unstable]` now due to the likelihood of being removed in the future if it is found to not be very useful. * copy_mut_lifetime - This function was also moved to `mem`, but had the same treatment as `copy_lifetime`. * copy_lifetime_vec - This function was removed because it is not used today, and its existence is not necessary with DST (copy_lifetime will suffice). In summary, the cast module was stripped down to these functions, and then the functions were moved to the `mem` module. transmute - #[unstable] transmute_copy - #[stable] forget - #[stable] copy_lifetime - #[unstable] copy_mut_lifetime - #[unstable] [breaking-change]
2014-05-09 12:34:51 -05:00
use std::str;
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
use syntax::abi;
rustc: Optimize reading metadata by 4x We were previously reading metadata via `ar p`, but as learned from rustdoc awhile back, spawning a process to do something is pretty slow. Turns out LLVM has an Archive class to read archives, but it cannot write archives. This commits adds bindings to the read-only version of the LLVM archive class (with a new type that only has a read() method), and then it uses this class when reading the metadata out of rlibs. When you put this in tandem of not compressing the metadata, reading the metadata is 4x faster than it used to be The timings I got for reading metadata from the respective libraries was: libstd-04ff901e-0.9-pre.dylib => 100ms libstd-04ff901e-0.9-pre.rlib => 23ms librustuv-7945354c-0.9-pre.dylib => 4ms librustuv-7945354c-0.9-pre.rlib => 1ms librustc-5b94a16f-0.9-pre.dylib => 87ms librustc-5b94a16f-0.9-pre.rlib => 35ms libextra-a6ebb16f-0.9-pre.dylib => 63ms libextra-a6ebb16f-0.9-pre.rlib => 15ms libsyntax-2e4c0458-0.9-pre.dylib => 86ms libsyntax-2e4c0458-0.9-pre.rlib => 22ms In order to always take advantage of these faster metadata read-times, I sort the files in filesearch based on whether they have an rlib extension or not (prefer all rlib files first). Overall, this halved the compile time for a `fn main() {}` crate from 0.185s to 0.095s on my system (when preferring dynamic linking). Reading metadata is still the slowest pass of the compiler at 0.035s, but it's getting pretty close to linking at 0.021s! The next best optimization is to just not copy the metadata from LLVM because that's the most expensive part of reading metadata right now.
2013-12-16 22:58:21 -06:00
pub static METADATA_FILENAME: &'static str = "rust.metadata.bin";
Store metadata separately in rlib files Right now whenever an rlib file is linked against, all of the metadata from the rlib is pulled in to the final staticlib or binary. The reason for this is that the metadata is currently stored in a section of the object file. Note that this is intentional for dynamic libraries in order to distribute metadata bundled with static libraries. This commit alters the situation for rlib libraries to instead store the metadata in a separate file in the archive. In doing so, when the archive is passed to the linker, none of the metadata will get pulled into the result executable. Furthermore, the metadata file is skipped when assembling rlibs into an archive. The snag in this implementation comes with multiple output formats. When generating a dylib, the metadata needs to be in the object file, but when generating an rlib this needs to be separate. In order to accomplish this, the metadata variable is inserted into an entirely separate LLVM Module which is then codegen'd into a different location (foo.metadata.o). This is then linked into dynamic libraries and silently ignored for rlib files. While changing how metadata is inserted into archives, I have also stopped compressing metadata when inserted into rlib files. We have wanted to stop compressing metadata, but the sections it creates in object file sections are apparently too large. Thankfully if it's just an arbitrary file it doesn't matter how large it is. I have seen massive reductions in executable sizes, as well as staticlib output sizes (to confirm that this is all working).
2013-12-03 19:41:01 -06:00
2014-03-05 08:36:01 -06:00
pub struct Archive<'a> {
sess: &'a Session,
dst: Path,
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
}
rustc: Optimize reading metadata by 4x We were previously reading metadata via `ar p`, but as learned from rustdoc awhile back, spawning a process to do something is pretty slow. Turns out LLVM has an Archive class to read archives, but it cannot write archives. This commits adds bindings to the read-only version of the LLVM archive class (with a new type that only has a read() method), and then it uses this class when reading the metadata out of rlibs. When you put this in tandem of not compressing the metadata, reading the metadata is 4x faster than it used to be The timings I got for reading metadata from the respective libraries was: libstd-04ff901e-0.9-pre.dylib => 100ms libstd-04ff901e-0.9-pre.rlib => 23ms librustuv-7945354c-0.9-pre.dylib => 4ms librustuv-7945354c-0.9-pre.rlib => 1ms librustc-5b94a16f-0.9-pre.dylib => 87ms librustc-5b94a16f-0.9-pre.rlib => 35ms libextra-a6ebb16f-0.9-pre.dylib => 63ms libextra-a6ebb16f-0.9-pre.rlib => 15ms libsyntax-2e4c0458-0.9-pre.dylib => 86ms libsyntax-2e4c0458-0.9-pre.rlib => 22ms In order to always take advantage of these faster metadata read-times, I sort the files in filesearch based on whether they have an rlib extension or not (prefer all rlib files first). Overall, this halved the compile time for a `fn main() {}` crate from 0.185s to 0.095s on my system (when preferring dynamic linking). Reading metadata is still the slowest pass of the compiler at 0.035s, but it's getting pretty close to linking at 0.021s! The next best optimization is to just not copy the metadata from LLVM because that's the most expensive part of reading metadata right now.
2013-12-16 22:58:21 -06:00
pub struct ArchiveRO {
ptr: ArchiveRef,
rustc: Optimize reading metadata by 4x We were previously reading metadata via `ar p`, but as learned from rustdoc awhile back, spawning a process to do something is pretty slow. Turns out LLVM has an Archive class to read archives, but it cannot write archives. This commits adds bindings to the read-only version of the LLVM archive class (with a new type that only has a read() method), and then it uses this class when reading the metadata out of rlibs. When you put this in tandem of not compressing the metadata, reading the metadata is 4x faster than it used to be The timings I got for reading metadata from the respective libraries was: libstd-04ff901e-0.9-pre.dylib => 100ms libstd-04ff901e-0.9-pre.rlib => 23ms librustuv-7945354c-0.9-pre.dylib => 4ms librustuv-7945354c-0.9-pre.rlib => 1ms librustc-5b94a16f-0.9-pre.dylib => 87ms librustc-5b94a16f-0.9-pre.rlib => 35ms libextra-a6ebb16f-0.9-pre.dylib => 63ms libextra-a6ebb16f-0.9-pre.rlib => 15ms libsyntax-2e4c0458-0.9-pre.dylib => 86ms libsyntax-2e4c0458-0.9-pre.rlib => 22ms In order to always take advantage of these faster metadata read-times, I sort the files in filesearch based on whether they have an rlib extension or not (prefer all rlib files first). Overall, this halved the compile time for a `fn main() {}` crate from 0.185s to 0.095s on my system (when preferring dynamic linking). Reading metadata is still the slowest pass of the compiler at 0.035s, but it's getting pretty close to linking at 0.021s! The next best optimization is to just not copy the metadata from LLVM because that's the most expensive part of reading metadata right now.
2013-12-16 22:58:21 -06:00
}
2014-03-05 08:36:01 -06:00
fn run_ar(sess: &Session, args: &str, cwd: Option<&Path>,
paths: &[&Path]) -> ProcessOutput {
let ar = get_ar_prog(sess);
let mut cmd = Command::new(ar.as_slice());
cmd.arg(args).args(paths);
debug!("{}", cmd);
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
match cwd {
Some(p) => {
cmd.cwd(p);
debug!("inside {}", p.display());
}
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
None => {}
}
match cmd.spawn() {
Ok(prog) => {
let o = prog.wait_with_output().unwrap();
if !o.status.success() {
sess.err(format!("{} failed with: {}",
cmd,
o.status).as_slice());
sess.note(format!("stdout ---\n{}",
str::from_utf8(o.output
.as_slice()).unwrap())
.as_slice());
sess.note(format!("stderr ---\n{}",
str::from_utf8(o.error
.as_slice()).unwrap())
.as_slice());
sess.abort_if_errors();
}
o
},
2014-01-29 20:42:19 -06:00
Err(e) => {
sess.err(format!("could not exec `{}`: {}", ar.as_slice(),
e).as_slice());
sess.abort_if_errors();
fail!("rustc::back::archive::run_ar() should not reach this point");
}
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
}
}
2014-03-05 08:36:01 -06:00
impl<'a> Archive<'a> {
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
/// Initializes a new static archive with the given object file
2014-03-05 08:36:01 -06:00
pub fn create<'b>(sess: &'a Session, dst: &'b Path,
initial_object: &'b Path) -> Archive<'a> {
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
run_ar(sess, "crus", None, [dst, initial_object]);
Archive { sess: sess, dst: dst.clone() }
}
/// Opens an existing static archive
2014-03-05 08:36:01 -06:00
pub fn open(sess: &'a Session, dst: Path) -> Archive<'a> {
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
assert!(dst.exists());
Archive { sess: sess, dst: dst }
}
/// Adds all of the contents of a native library to this archive. This will
/// search in the relevant locations for a library named `name`.
2014-01-29 20:42:19 -06:00
pub fn add_native_library(&mut self, name: &str) -> io::IoResult<()> {
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
let location = self.find_library(name);
2014-01-29 20:42:19 -06:00
self.add_archive(&location, name, [])
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
}
/// Adds all of the contents of the rlib at the specified path to this
/// archive.
Implement LTO This commit implements LTO for rust leveraging LLVM's passes. What this means is: * When compiling an rlib, in addition to insdering foo.o into the archive, also insert foo.bc (the LLVM bytecode) of the optimized module. * When the compiler detects the -Z lto option, it will attempt to perform LTO on a staticlib or binary output. The compiler will emit an error if a dylib or rlib output is being generated. * The actual act of performing LTO is as follows: 1. Force all upstream libraries to have an rlib version available. 2. Load the bytecode of each upstream library from the rlib. 3. Link all this bytecode into the current LLVM module (just using llvm apis) 4. Run an internalization pass which internalizes all symbols except those found reachable for the local crate of compilation. 5. Run the LLVM LTO pass manager over this entire module 6a. If assembling an archive, then add all upstream rlibs into the output archive. This ignores all of the object/bitcode/metadata files rust generated and placed inside the rlibs. 6b. If linking a binary, create copies of all upstream rlibs, remove the rust-generated object-file, and then link everything as usual. As I have explained in #10741, this process is excruciatingly slow, so this is *not* turned on by default, and it is also why I have decided to hide it behind a -Z flag for now. The good news is that the binary sizes are about as small as they can be as a result of LTO, so it's definitely working. Closes #10741 Closes #10740
2013-12-03 01:19:29 -06:00
///
/// This ignores adding the bytecode from the rlib, and if LTO is enabled
/// then the object file also isn't added.
2014-01-29 20:42:19 -06:00
pub fn add_rlib(&mut self, rlib: &Path, name: &str,
lto: bool) -> io::IoResult<()> {
Implement LTO This commit implements LTO for rust leveraging LLVM's passes. What this means is: * When compiling an rlib, in addition to insdering foo.o into the archive, also insert foo.bc (the LLVM bytecode) of the optimized module. * When the compiler detects the -Z lto option, it will attempt to perform LTO on a staticlib or binary output. The compiler will emit an error if a dylib or rlib output is being generated. * The actual act of performing LTO is as follows: 1. Force all upstream libraries to have an rlib version available. 2. Load the bytecode of each upstream library from the rlib. 3. Link all this bytecode into the current LLVM module (just using llvm apis) 4. Run an internalization pass which internalizes all symbols except those found reachable for the local crate of compilation. 5. Run the LLVM LTO pass manager over this entire module 6a. If assembling an archive, then add all upstream rlibs into the output archive. This ignores all of the object/bitcode/metadata files rust generated and placed inside the rlibs. 6b. If linking a binary, create copies of all upstream rlibs, remove the rust-generated object-file, and then link everything as usual. As I have explained in #10741, this process is excruciatingly slow, so this is *not* turned on by default, and it is also why I have decided to hide it behind a -Z flag for now. The good news is that the binary sizes are about as small as they can be as a result of LTO, so it's definitely working. Closes #10741 Closes #10740
2013-12-03 01:19:29 -06:00
let object = format!("{}.o", name);
let bytecode = format!("{}.bytecode.deflate", name);
let mut ignore = vec!(bytecode.as_slice(), METADATA_FILENAME);
Implement LTO This commit implements LTO for rust leveraging LLVM's passes. What this means is: * When compiling an rlib, in addition to insdering foo.o into the archive, also insert foo.bc (the LLVM bytecode) of the optimized module. * When the compiler detects the -Z lto option, it will attempt to perform LTO on a staticlib or binary output. The compiler will emit an error if a dylib or rlib output is being generated. * The actual act of performing LTO is as follows: 1. Force all upstream libraries to have an rlib version available. 2. Load the bytecode of each upstream library from the rlib. 3. Link all this bytecode into the current LLVM module (just using llvm apis) 4. Run an internalization pass which internalizes all symbols except those found reachable for the local crate of compilation. 5. Run the LLVM LTO pass manager over this entire module 6a. If assembling an archive, then add all upstream rlibs into the output archive. This ignores all of the object/bitcode/metadata files rust generated and placed inside the rlibs. 6b. If linking a binary, create copies of all upstream rlibs, remove the rust-generated object-file, and then link everything as usual. As I have explained in #10741, this process is excruciatingly slow, so this is *not* turned on by default, and it is also why I have decided to hide it behind a -Z flag for now. The good news is that the binary sizes are about as small as they can be as a result of LTO, so it's definitely working. Closes #10741 Closes #10740
2013-12-03 01:19:29 -06:00
if lto {
ignore.push(object.as_slice());
}
self.add_archive(rlib, name, ignore.as_slice())
Store metadata separately in rlib files Right now whenever an rlib file is linked against, all of the metadata from the rlib is pulled in to the final staticlib or binary. The reason for this is that the metadata is currently stored in a section of the object file. Note that this is intentional for dynamic libraries in order to distribute metadata bundled with static libraries. This commit alters the situation for rlib libraries to instead store the metadata in a separate file in the archive. In doing so, when the archive is passed to the linker, none of the metadata will get pulled into the result executable. Furthermore, the metadata file is skipped when assembling rlibs into an archive. The snag in this implementation comes with multiple output formats. When generating a dylib, the metadata needs to be in the object file, but when generating an rlib this needs to be separate. In order to accomplish this, the metadata variable is inserted into an entirely separate LLVM Module which is then codegen'd into a different location (foo.metadata.o). This is then linked into dynamic libraries and silently ignored for rlib files. While changing how metadata is inserted into archives, I have also stopped compressing metadata when inserted into rlib files. We have wanted to stop compressing metadata, but the sections it creates in object file sections are apparently too large. Thankfully if it's just an arbitrary file it doesn't matter how large it is. I have seen massive reductions in executable sizes, as well as staticlib output sizes (to confirm that this is all working).
2013-12-03 19:41:01 -06:00
}
/// Adds an arbitrary file to this archive
pub fn add_file(&mut self, file: &Path, has_symbols: bool) {
let cmd = if has_symbols {"r"} else {"rS"};
run_ar(self.sess, cmd, None, [&self.dst, file]);
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
}
Implement LTO This commit implements LTO for rust leveraging LLVM's passes. What this means is: * When compiling an rlib, in addition to insdering foo.o into the archive, also insert foo.bc (the LLVM bytecode) of the optimized module. * When the compiler detects the -Z lto option, it will attempt to perform LTO on a staticlib or binary output. The compiler will emit an error if a dylib or rlib output is being generated. * The actual act of performing LTO is as follows: 1. Force all upstream libraries to have an rlib version available. 2. Load the bytecode of each upstream library from the rlib. 3. Link all this bytecode into the current LLVM module (just using llvm apis) 4. Run an internalization pass which internalizes all symbols except those found reachable for the local crate of compilation. 5. Run the LLVM LTO pass manager over this entire module 6a. If assembling an archive, then add all upstream rlibs into the output archive. This ignores all of the object/bitcode/metadata files rust generated and placed inside the rlibs. 6b. If linking a binary, create copies of all upstream rlibs, remove the rust-generated object-file, and then link everything as usual. As I have explained in #10741, this process is excruciatingly slow, so this is *not* turned on by default, and it is also why I have decided to hide it behind a -Z flag for now. The good news is that the binary sizes are about as small as they can be as a result of LTO, so it's definitely working. Closes #10741 Closes #10740
2013-12-03 01:19:29 -06:00
/// Removes a file from this archive
pub fn remove_file(&mut self, file: &str) {
run_ar(self.sess, "d", None, [&self.dst, &Path::new(file)]);
}
/// Updates all symbols in the archive (runs 'ar s' over it)
pub fn update_symbols(&mut self) {
run_ar(self.sess, "s", None, [&self.dst]);
}
/// Lists all files in an archive
pub fn files(&self) -> Vec<String> {
Implement LTO This commit implements LTO for rust leveraging LLVM's passes. What this means is: * When compiling an rlib, in addition to insdering foo.o into the archive, also insert foo.bc (the LLVM bytecode) of the optimized module. * When the compiler detects the -Z lto option, it will attempt to perform LTO on a staticlib or binary output. The compiler will emit an error if a dylib or rlib output is being generated. * The actual act of performing LTO is as follows: 1. Force all upstream libraries to have an rlib version available. 2. Load the bytecode of each upstream library from the rlib. 3. Link all this bytecode into the current LLVM module (just using llvm apis) 4. Run an internalization pass which internalizes all symbols except those found reachable for the local crate of compilation. 5. Run the LLVM LTO pass manager over this entire module 6a. If assembling an archive, then add all upstream rlibs into the output archive. This ignores all of the object/bitcode/metadata files rust generated and placed inside the rlibs. 6b. If linking a binary, create copies of all upstream rlibs, remove the rust-generated object-file, and then link everything as usual. As I have explained in #10741, this process is excruciatingly slow, so this is *not* turned on by default, and it is also why I have decided to hide it behind a -Z flag for now. The good news is that the binary sizes are about as small as they can be as a result of LTO, so it's definitely working. Closes #10741 Closes #10740
2013-12-03 01:19:29 -06:00
let output = run_ar(self.sess, "t", None, [&self.dst]);
let output = str::from_utf8(output.output.as_slice()).unwrap();
// use lines_any because windows delimits output with `\r\n` instead of
// just `\n`
output.lines_any().map(|s| s.to_string()).collect()
Implement LTO This commit implements LTO for rust leveraging LLVM's passes. What this means is: * When compiling an rlib, in addition to insdering foo.o into the archive, also insert foo.bc (the LLVM bytecode) of the optimized module. * When the compiler detects the -Z lto option, it will attempt to perform LTO on a staticlib or binary output. The compiler will emit an error if a dylib or rlib output is being generated. * The actual act of performing LTO is as follows: 1. Force all upstream libraries to have an rlib version available. 2. Load the bytecode of each upstream library from the rlib. 3. Link all this bytecode into the current LLVM module (just using llvm apis) 4. Run an internalization pass which internalizes all symbols except those found reachable for the local crate of compilation. 5. Run the LLVM LTO pass manager over this entire module 6a. If assembling an archive, then add all upstream rlibs into the output archive. This ignores all of the object/bitcode/metadata files rust generated and placed inside the rlibs. 6b. If linking a binary, create copies of all upstream rlibs, remove the rust-generated object-file, and then link everything as usual. As I have explained in #10741, this process is excruciatingly slow, so this is *not* turned on by default, and it is also why I have decided to hide it behind a -Z flag for now. The good news is that the binary sizes are about as small as they can be as a result of LTO, so it's definitely working. Closes #10741 Closes #10740
2013-12-03 01:19:29 -06:00
}
2014-01-29 20:42:19 -06:00
fn add_archive(&mut self, archive: &Path, name: &str,
skip: &[&str]) -> io::IoResult<()> {
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
let loc = TempDir::new("rsar").unwrap();
// First, extract the contents of the archive to a temporary directory
let archive = os::make_absolute(archive);
run_ar(self.sess, "x", Some(loc.path()), [&archive]);
// Next, we must rename all of the inputs to "guaranteed unique names".
// The reason for this is that archives are keyed off the name of the
// files, so if two files have the same name they will override one
// another in the archive (bad).
Implement LTO This commit implements LTO for rust leveraging LLVM's passes. What this means is: * When compiling an rlib, in addition to insdering foo.o into the archive, also insert foo.bc (the LLVM bytecode) of the optimized module. * When the compiler detects the -Z lto option, it will attempt to perform LTO on a staticlib or binary output. The compiler will emit an error if a dylib or rlib output is being generated. * The actual act of performing LTO is as follows: 1. Force all upstream libraries to have an rlib version available. 2. Load the bytecode of each upstream library from the rlib. 3. Link all this bytecode into the current LLVM module (just using llvm apis) 4. Run an internalization pass which internalizes all symbols except those found reachable for the local crate of compilation. 5. Run the LLVM LTO pass manager over this entire module 6a. If assembling an archive, then add all upstream rlibs into the output archive. This ignores all of the object/bitcode/metadata files rust generated and placed inside the rlibs. 6b. If linking a binary, create copies of all upstream rlibs, remove the rust-generated object-file, and then link everything as usual. As I have explained in #10741, this process is excruciatingly slow, so this is *not* turned on by default, and it is also why I have decided to hide it behind a -Z flag for now. The good news is that the binary sizes are about as small as they can be as a result of LTO, so it's definitely working. Closes #10741 Closes #10740
2013-12-03 01:19:29 -06:00
//
// We skip any files explicitly desired for skipping, and we also skip
// all SYMDEF files as these are just magical placeholders which get
// re-created when we make a new archive anyway.
let files = try!(fs::readdir(loc.path()));
let mut inputs = Vec::new();
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
for file in files.iter() {
let filename = file.filename_str().unwrap();
Store metadata separately in rlib files Right now whenever an rlib file is linked against, all of the metadata from the rlib is pulled in to the final staticlib or binary. The reason for this is that the metadata is currently stored in a section of the object file. Note that this is intentional for dynamic libraries in order to distribute metadata bundled with static libraries. This commit alters the situation for rlib libraries to instead store the metadata in a separate file in the archive. In doing so, when the archive is passed to the linker, none of the metadata will get pulled into the result executable. Furthermore, the metadata file is skipped when assembling rlibs into an archive. The snag in this implementation comes with multiple output formats. When generating a dylib, the metadata needs to be in the object file, but when generating an rlib this needs to be separate. In order to accomplish this, the metadata variable is inserted into an entirely separate LLVM Module which is then codegen'd into a different location (foo.metadata.o). This is then linked into dynamic libraries and silently ignored for rlib files. While changing how metadata is inserted into archives, I have also stopped compressing metadata when inserted into rlib files. We have wanted to stop compressing metadata, but the sections it creates in object file sections are apparently too large. Thankfully if it's just an arbitrary file it doesn't matter how large it is. I have seen massive reductions in executable sizes, as well as staticlib output sizes (to confirm that this is all working).
2013-12-03 19:41:01 -06:00
if skip.iter().any(|s| *s == filename) { continue }
Implement LTO This commit implements LTO for rust leveraging LLVM's passes. What this means is: * When compiling an rlib, in addition to insdering foo.o into the archive, also insert foo.bc (the LLVM bytecode) of the optimized module. * When the compiler detects the -Z lto option, it will attempt to perform LTO on a staticlib or binary output. The compiler will emit an error if a dylib or rlib output is being generated. * The actual act of performing LTO is as follows: 1. Force all upstream libraries to have an rlib version available. 2. Load the bytecode of each upstream library from the rlib. 3. Link all this bytecode into the current LLVM module (just using llvm apis) 4. Run an internalization pass which internalizes all symbols except those found reachable for the local crate of compilation. 5. Run the LLVM LTO pass manager over this entire module 6a. If assembling an archive, then add all upstream rlibs into the output archive. This ignores all of the object/bitcode/metadata files rust generated and placed inside the rlibs. 6b. If linking a binary, create copies of all upstream rlibs, remove the rust-generated object-file, and then link everything as usual. As I have explained in #10741, this process is excruciatingly slow, so this is *not* turned on by default, and it is also why I have decided to hide it behind a -Z flag for now. The good news is that the binary sizes are about as small as they can be as a result of LTO, so it's definitely working. Closes #10741 Closes #10740
2013-12-03 01:19:29 -06:00
if filename.contains(".SYMDEF") { continue }
let filename = format!("r-{}-{}", name, filename);
// LLDB (as mentioned in back::link) crashes on filenames of exactly
// 16 bytes in length. If we're including an object file with
// exactly 16-bytes of characters, give it some prefix so that it's
// not 16 bytes.
let filename = if filename.len() == 16 {
format!("lldb-fix-{}", filename)
} else {
filename
};
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
let new_filename = file.with_filename(filename);
try!(fs::rename(file, &new_filename));
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
inputs.push(new_filename);
}
if inputs.len() == 0 { return Ok(()) }
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
// Finally, add all the renamed files to this archive
let mut args = vec!(&self.dst);
args.extend(inputs.iter());
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
run_ar(self.sess, "r", None, args.as_slice());
2014-01-29 20:42:19 -06:00
Ok(())
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
}
fn find_library(&self, name: &str) -> Path {
let (osprefix, osext) = match self.sess.targ_cfg.os {
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
abi::OsWin32 => ("", "lib"), _ => ("lib", "a"),
};
// On Windows, static libraries sometimes show up as libfoo.a and other
// times show up as foo.lib
let oslibname = format!("{}{}.{}", osprefix, name, osext);
let unixlibname = format!("lib{}.a", name);
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
let mut rustpath = filesearch::rust_path();
rustpath.push(self.sess.target_filesearch().get_lib_path());
2014-03-20 21:49:20 -05:00
let search = self.sess.opts.addl_lib_search_paths.borrow();
for path in search.iter().chain(rustpath.iter()) {
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
debug!("looking for {} inside {}", name, path.display());
let test = path.join(oslibname.as_slice());
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
if test.exists() { return test }
if oslibname != unixlibname {
let test = path.join(unixlibname.as_slice());
if test.exists() { return test }
}
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
}
self.sess.fatal(format!("could not find native static library `{}`, \
perhaps an -L flag is missing?",
name).as_slice());
Add generation of static libraries to rustc This commit implements the support necessary for generating both intermediate and result static rust libraries. This is an implementation of my thoughts in https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html. When compiling a library, we still retain the "lib" option, although now there are "rlib", "staticlib", and "dylib" as options for crate_type (and these are stackable). The idea of "lib" is to generate the "compiler default" instead of having too choose (although all are interchangeable). For now I have left the "complier default" to be a dynamic library for size reasons. Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a dynamic object. I chose this for size reasons, but also because you're probably not going to be embedding the rustc compiler anywhere any time soon. Other than the options outlined above, there are a few defaults/preferences that are now opinionated in the compiler: * If both a .dylib and .rlib are found for a rust library, the compiler will prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option * If generating a "lib", the compiler will generate a dynamic library. This is overridable by explicitly saying what flavor you'd like (rlib, staticlib, dylib). * If no options are passed to the command line, and no crate_type is found in the destination crate, then an executable is generated With this change, you can successfully build a rust program with 0 dynamic dependencies on rust libraries. There is still a dynamic dependency on librustrt, but I plan on removing that in a subsequent commit. This change includes no tests just yet. Our current testing infrastructure/harnesses aren't very amenable to doing flavorful things with linking, so I'm planning on adding a new mode of testing which I believe belongs as a separate commit. Closes #552
2013-11-15 16:03:29 -06:00
}
}
rustc: Optimize reading metadata by 4x We were previously reading metadata via `ar p`, but as learned from rustdoc awhile back, spawning a process to do something is pretty slow. Turns out LLVM has an Archive class to read archives, but it cannot write archives. This commits adds bindings to the read-only version of the LLVM archive class (with a new type that only has a read() method), and then it uses this class when reading the metadata out of rlibs. When you put this in tandem of not compressing the metadata, reading the metadata is 4x faster than it used to be The timings I got for reading metadata from the respective libraries was: libstd-04ff901e-0.9-pre.dylib => 100ms libstd-04ff901e-0.9-pre.rlib => 23ms librustuv-7945354c-0.9-pre.dylib => 4ms librustuv-7945354c-0.9-pre.rlib => 1ms librustc-5b94a16f-0.9-pre.dylib => 87ms librustc-5b94a16f-0.9-pre.rlib => 35ms libextra-a6ebb16f-0.9-pre.dylib => 63ms libextra-a6ebb16f-0.9-pre.rlib => 15ms libsyntax-2e4c0458-0.9-pre.dylib => 86ms libsyntax-2e4c0458-0.9-pre.rlib => 22ms In order to always take advantage of these faster metadata read-times, I sort the files in filesearch based on whether they have an rlib extension or not (prefer all rlib files first). Overall, this halved the compile time for a `fn main() {}` crate from 0.185s to 0.095s on my system (when preferring dynamic linking). Reading metadata is still the slowest pass of the compiler at 0.035s, but it's getting pretty close to linking at 0.021s! The next best optimization is to just not copy the metadata from LLVM because that's the most expensive part of reading metadata right now.
2013-12-16 22:58:21 -06:00
impl ArchiveRO {
/// Opens a static archive for read-only purposes. This is more optimized
/// than the `open` method because it uses LLVM's internal `Archive` class
/// rather than shelling out to `ar` for everything.
///
/// If this archive is used with a mutable method, then an error will be
/// raised.
pub fn open(dst: &Path) -> Option<ArchiveRO> {
unsafe {
let ar = dst.with_c_str(|dst| {
llvm::LLVMRustOpenArchive(dst)
});
if ar.is_null() {
None
} else {
Some(ArchiveRO { ptr: ar })
}
}
}
/// Reads a file in the archive
rustc: Optimize reading metadata by 4x We were previously reading metadata via `ar p`, but as learned from rustdoc awhile back, spawning a process to do something is pretty slow. Turns out LLVM has an Archive class to read archives, but it cannot write archives. This commits adds bindings to the read-only version of the LLVM archive class (with a new type that only has a read() method), and then it uses this class when reading the metadata out of rlibs. When you put this in tandem of not compressing the metadata, reading the metadata is 4x faster than it used to be The timings I got for reading metadata from the respective libraries was: libstd-04ff901e-0.9-pre.dylib => 100ms libstd-04ff901e-0.9-pre.rlib => 23ms librustuv-7945354c-0.9-pre.dylib => 4ms librustuv-7945354c-0.9-pre.rlib => 1ms librustc-5b94a16f-0.9-pre.dylib => 87ms librustc-5b94a16f-0.9-pre.rlib => 35ms libextra-a6ebb16f-0.9-pre.dylib => 63ms libextra-a6ebb16f-0.9-pre.rlib => 15ms libsyntax-2e4c0458-0.9-pre.dylib => 86ms libsyntax-2e4c0458-0.9-pre.rlib => 22ms In order to always take advantage of these faster metadata read-times, I sort the files in filesearch based on whether they have an rlib extension or not (prefer all rlib files first). Overall, this halved the compile time for a `fn main() {}` crate from 0.185s to 0.095s on my system (when preferring dynamic linking). Reading metadata is still the slowest pass of the compiler at 0.035s, but it's getting pretty close to linking at 0.021s! The next best optimization is to just not copy the metadata from LLVM because that's the most expensive part of reading metadata right now.
2013-12-16 22:58:21 -06:00
pub fn read<'a>(&'a self, file: &str) -> Option<&'a [u8]> {
unsafe {
let mut size = 0 as libc::size_t;
let ptr = file.with_c_str(|file| {
llvm::LLVMRustArchiveReadSection(self.ptr, file, &mut size)
});
if ptr.is_null() {
None
} else {
core: Remove the cast module This commit revisits the `cast` module in libcore and libstd, and scrutinizes all functions inside of it. The result was to remove the `cast` module entirely, folding all functionality into the `mem` module. Specifically, this is the fate of each function in the `cast` module. * transmute - This function was moved to `mem`, but it is now marked as #[unstable]. This is due to planned changes to the `transmute` function and how it can be invoked (see the #[unstable] comment). For more information, see RFC 5 and #12898 * transmute_copy - This function was moved to `mem`, with clarification that is is not an error to invoke it with T/U that are different sizes, but rather that it is strongly discouraged. This function is now #[stable] * forget - This function was moved to `mem` and marked #[stable] * bump_box_refcount - This function was removed due to the deprecation of managed boxes as well as its questionable utility. * transmute_mut - This function was previously deprecated, and removed as part of this commit. * transmute_mut_unsafe - This function doesn't serve much of a purpose when it can be achieved with an `as` in safe code, so it was removed. * transmute_lifetime - This function was removed because it is likely a strong indication that code is incorrect in the first place. * transmute_mut_lifetime - This function was removed for the same reasons as `transmute_lifetime` * copy_lifetime - This function was moved to `mem`, but it is marked `#[unstable]` now due to the likelihood of being removed in the future if it is found to not be very useful. * copy_mut_lifetime - This function was also moved to `mem`, but had the same treatment as `copy_lifetime`. * copy_lifetime_vec - This function was removed because it is not used today, and its existence is not necessary with DST (copy_lifetime will suffice). In summary, the cast module was stripped down to these functions, and then the functions were moved to the `mem` module. transmute - #[unstable] transmute_copy - #[stable] forget - #[stable] copy_lifetime - #[unstable] copy_mut_lifetime - #[unstable] [breaking-change]
2014-05-09 12:34:51 -05:00
Some(mem::transmute(raw::Slice {
rustc: Optimize reading metadata by 4x We were previously reading metadata via `ar p`, but as learned from rustdoc awhile back, spawning a process to do something is pretty slow. Turns out LLVM has an Archive class to read archives, but it cannot write archives. This commits adds bindings to the read-only version of the LLVM archive class (with a new type that only has a read() method), and then it uses this class when reading the metadata out of rlibs. When you put this in tandem of not compressing the metadata, reading the metadata is 4x faster than it used to be The timings I got for reading metadata from the respective libraries was: libstd-04ff901e-0.9-pre.dylib => 100ms libstd-04ff901e-0.9-pre.rlib => 23ms librustuv-7945354c-0.9-pre.dylib => 4ms librustuv-7945354c-0.9-pre.rlib => 1ms librustc-5b94a16f-0.9-pre.dylib => 87ms librustc-5b94a16f-0.9-pre.rlib => 35ms libextra-a6ebb16f-0.9-pre.dylib => 63ms libextra-a6ebb16f-0.9-pre.rlib => 15ms libsyntax-2e4c0458-0.9-pre.dylib => 86ms libsyntax-2e4c0458-0.9-pre.rlib => 22ms In order to always take advantage of these faster metadata read-times, I sort the files in filesearch based on whether they have an rlib extension or not (prefer all rlib files first). Overall, this halved the compile time for a `fn main() {}` crate from 0.185s to 0.095s on my system (when preferring dynamic linking). Reading metadata is still the slowest pass of the compiler at 0.035s, but it's getting pretty close to linking at 0.021s! The next best optimization is to just not copy the metadata from LLVM because that's the most expensive part of reading metadata right now.
2013-12-16 22:58:21 -06:00
data: ptr,
len: size as uint,
}))
}
}
}
}
impl Drop for ArchiveRO {
fn drop(&mut self) {
unsafe {
llvm::LLVMRustDestroyArchive(self.ptr);
}
}
}