This uses quite a bit of unsafe code for speed and failure safety, and allocates `2*n` temporary storage.
[Performance](https://gist.github.com/huonw/5547f2478380288a28c2):
| n | new | priority_queue | quick3 |
|-------:|---------:|---------------:|---------:|
| 5 | 200 | 155 | 106 |
| 100 | 6490 | 8750 | 5810 |
| 10000 | 1300000 | 1790000 | 1060000 |
| 100000 | 16700000 | 23600000 | 12700000 |
| sorted | 520000 | 1380000 | 53900000 |
| trend | 1310000 | 1690000 | 1100000 |
(The times are in nanoseconds, having subtracted the set-up time (i.e. the `just_generate` bench target).)
I imagine that there is still significant room for improvement, particularly because both priority_queue and quick3 are doing a static call via `Ord` or `TotalOrd` for the comparisons, while this is using a (boxed) closure.
Also, this code does not `clone`, unlike `quick_sort3`; and is stable, unlike both of the others.
We were previously reading metadata via `ar p`, but as learned from rustdoc
awhile back, spawning a process to do something is pretty slow. Turns out LLVM
has an Archive class to read archives, but it cannot write archives.
This commits adds bindings to the read-only version of the LLVM archive class
(with a new type that only has a read() method), and then it uses this class
when reading the metadata out of rlibs. When you put this in tandem of not
compressing the metadata, reading the metadata is 4x faster than it used to be
The timings I got for reading metadata from the respective libraries was:
libstd-04ff901e-0.9-pre.dylib => 100ms
libstd-04ff901e-0.9-pre.rlib => 23ms
librustuv-7945354c-0.9-pre.dylib => 4ms
librustuv-7945354c-0.9-pre.rlib => 1ms
librustc-5b94a16f-0.9-pre.dylib => 87ms
librustc-5b94a16f-0.9-pre.rlib => 35ms
libextra-a6ebb16f-0.9-pre.dylib => 63ms
libextra-a6ebb16f-0.9-pre.rlib => 15ms
libsyntax-2e4c0458-0.9-pre.dylib => 86ms
libsyntax-2e4c0458-0.9-pre.rlib => 22ms
In order to always take advantage of these faster metadata read-times, I sort
the files in filesearch based on whether they have an rlib extension or not
(prefer all rlib files first).
Overall, this halved the compile time for a `fn main() {}` crate from 0.185s to
0.095s on my system (when preferring dynamic linking). Reading metadata is still
the slowest pass of the compiler at 0.035s, but it's getting pretty close to
linking at 0.021s! The next best optimization is to just not copy the metadata
from LLVM because that's the most expensive part of reading metadata right now.
We were previously reading metadata via `ar p`, but as learned from rustdoc
awhile back, spawning a process to do something is pretty slow. Turns out LLVM
has an Archive class to read archives, but it cannot write archives.
This commits adds bindings to the read-only version of the LLVM archive class
(with a new type that only has a read() method), and then it uses this class
when reading the metadata out of rlibs. When you put this in tandem of not
compressing the metadata, reading the metadata is 4x faster than it used to be
The timings I got for reading metadata from the respective libraries was:
libstd-04ff901e-0.9-pre.dylib => 100ms
libstd-04ff901e-0.9-pre.rlib => 23ms
librustuv-7945354c-0.9-pre.dylib => 4ms
librustuv-7945354c-0.9-pre.rlib => 1ms
librustc-5b94a16f-0.9-pre.dylib => 87ms
librustc-5b94a16f-0.9-pre.rlib => 35ms
libextra-a6ebb16f-0.9-pre.dylib => 63ms
libextra-a6ebb16f-0.9-pre.rlib => 15ms
libsyntax-2e4c0458-0.9-pre.dylib => 86ms
libsyntax-2e4c0458-0.9-pre.rlib => 22ms
In order to always take advantage of these faster metadata read-times, I sort
the files in filesearch based on whether they have an rlib extension or not
(prefer all rlib files first).
Overall, this halved the compile time for a `fn main() {}` crate from 0.185s to
0.095s on my system (when preferring dynamic linking). Reading metadata is still
the slowest pass of the compiler at 0.035s, but it's getting pretty close to
linking at 0.021s! The next best optimization is to just not copy the metadata
from LLVM because that's the most expensive part of reading metadata right now.
Now that the metadata is an owned value with a lifetime of a borrowed byte
slice, it's possible to have future optimizations where the metadata doesn't
need to be copied around (very expensive operation).
Now that the metadata is an owned value with a lifetime of a borrowed byte
slice, it's possible to have future optimizations where the metadata doesn't
need to be copied around (very expensive operation).
If it's a trait method, this checks the stability attribute of the
method inside the trait definition. Otherwise, it checks the method
implementation itself.
Close#8961.
If it's a trait method, this checks the stability attribute of the
method inside the trait definition. Otherwise, it checks the method
implementation itself.
This replaces the link meta attributes with a pkgid attribute and uses a hash
of this as the crate hash. This makes the crate hash computable by things
other than the Rust compiler. It also switches the hash function ot SHA1 since
that is much more likely to be available in shell, Python, etc than SipHash.
Fixes#10188, #8523.
Right now whenever an rlib file is linked against, all of the metadata from the
rlib is pulled in to the final staticlib or binary. The reason for this is that
the metadata is currently stored in a section of the object file. Note that this
is intentional for dynamic libraries in order to distribute metadata bundled
with static libraries.
This commit alters the situation for rlib libraries to instead store the
metadata in a separate file in the archive. In doing so, when the archive is
passed to the linker, none of the metadata will get pulled into the result
executable. Furthermore, the metadata file is skipped when assembling rlibs into
an archive.
The snag in this implementation comes with multiple output formats. When
generating a dylib, the metadata needs to be in the object file, but when
generating an rlib this needs to be separate. In order to accomplish this, the
metadata variable is inserted into an entirely separate LLVM Module which is
then codegen'd into a different location (foo.metadata.o). This is then linked
into dynamic libraries and silently ignored for rlib files.
While changing how metadata is inserted into archives, I have also stopped
compressing metadata when inserted into rlib files. We have wanted to stop
compressing metadata, but the sections it creates in object file sections are
apparently too large. Thankfully if it's just an arbitrary file it doesn't
matter how large it is.
I have seen massive reductions in executable sizes, as well as staticlib output
sizes (to confirm that this is all working).
This reverts commit c54427ddfb.
Leave the #[ignores] in that were added to rustpkg tests.
Conflicts:
src/librustc/driver/driver.rs
src/librustc/metadata/creader.rs
This function had type &[u8] -> ~str, i.e. it allocates a string
internally, even though the non-allocating version that take &[u8] ->
&str and ~[u8] -> ~str are all that is necessary in most circumstances.
**Note**: I only tested on top of my #10670 PR, size reductions come from both change sets.
With this, [more enums are shrinked](https://gist.github.com/eddyb/08fef0dfc6ff54e890bc), the most significant one being `ast_node`, from 104 bytes (master) to 96 (#10670) and now to 32 bytes.
My own testcase requires **200MB** less when compiling (not including the other **200MB** gained in #10670), and rustc-stage2 is down by about **130MB**.
I believe there is more to gain by fiddling with the enums' layouts.
This adds support to link to OSX frameworks via the new link attribute when
using `kind = "framework"`. It is a compiler error to request linkage to a
framework when the target is not macos because other platforms don't support
frameworks.
Closes#2023
This commit implements the support necessary for generating both intermediate
and result static rust libraries. This is an implementation of my thoughts in
https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html.
When compiling a library, we still retain the "lib" option, although now there
are "rlib", "staticlib", and "dylib" as options for crate_type (and these are
stackable). The idea of "lib" is to generate the "compiler default" instead of
having too choose (although all are interchangeable). For now I have left the
"complier default" to be a dynamic library for size reasons.
Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an
rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a
dynamic object. I chose this for size reasons, but also because you're probably
not going to be embedding the rustc compiler anywhere any time soon.
Other than the options outlined above, there are a few defaults/preferences that
are now opinionated in the compiler:
* If both a .dylib and .rlib are found for a rust library, the compiler will
prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option
* If generating a "lib", the compiler will generate a dynamic library. This is
overridable by explicitly saying what flavor you'd like (rlib, staticlib,
dylib).
* If no options are passed to the command line, and no crate_type is found in
the destination crate, then an executable is generated
With this change, you can successfully build a rust program with 0 dynamic
dependencies on rust libraries. There is still a dynamic dependency on
librustrt, but I plan on removing that in a subsequent commit.
This change includes no tests just yet. Our current testing
infrastructure/harnesses aren't very amenable to doing flavorful things with
linking, so I'm planning on adding a new mode of testing which I believe belongs
as a separate commit.
Closes#552
### Rationale
There is no reason to support more than 2³² nodes or names at this moment, as compiling something that big (even without considering the quadratic space usage of some analysis passes) would take at least **64GB**.
Meanwhile, some can't (or barely can) compile rustc because it requires almost **1.5GB**.
### Potential problems
Can someone confirm this doesn't affect metadata (de)serialization? I can't tell myself, I know nothing about it.
### Results
Some structures have a size reduction of 25% to 50%: [before](https://gist.github.com/luqmana/3a82a51fa9c86d9191fa) - [after](https://gist.github.com/eddyb/5a75f8973d3d8018afd3).
Sadly, there isn't a massive change in the memory used for compiling stage2 librustc (it doesn't go over **1.4GB** as [before](http://huonw.github.io/isrustfastyet/mem/), but I can barely see the difference).
However, my own testcase (previously peaking at **1.6GB** in typeck) shows a reduction of **200**-**400MB**.
Fully support multiple lifetime parameters on types and elsewhere, removing special treatment for `'self`. I am submitting this a touch early in that I plan to push a new commit with more tests specifically targeting types with multiple lifetime parameters -- but the current code bootstraps and passes `make check`.
Fixes#4846
This renames the `file` module to `fs` because that more accurately describes
its current purpose (manipulating the filesystem, not just files).
Additionally, this adds an UnstableFileStat structure as a nested structure of
FileStat to signify that the fields should not be depended on. The structure is
currently flagged with #[unstable], but it's unlikely that it has much meaning.
Closes#10241
This commit moves all thread-blocking I/O functions from the std::os module.
Their replacements can be found in either std::rt::io::file or in a hidden
"old_os" module inside of native::file. I didn't want to outright delete these
functions because they have a lot of special casing learned over time for each
OS/platform, and I imagine that these will someday get integrated into a
blocking implementation of IoFactory. For now, they're moved to a private module
to prevent bitrot and still have tests to ensure that they work.
I've also expanded the extensions to a few more methods defined on Path, most of
which were previously defined in std::os but now have non-thread-blocking
implementations as part of using the current IoFactory.
The api of io::file is in flux, but I plan on changing it in the next commit as
well.
Closes#10057
This is a fairly brittle modle that doesn't scale well across many crates. It's
unlikely that all of the downstream crates will have all of the original native
dependencies of all the upstream crates. In the case that FFI functions are
reachable, then it should be the responsibility of the downstream crate to link
against the correct library, or the upstream crate should prevent the functions
from being reachable.
These methods are all excellent candidates for default methods, so there's no need to require extra imports of various traits. Additionally, this was able to remove all the weird underscores after the method names. Yay!
Delete the following API functions:
- set_dirname()
- with_dirname()
- set_filestem()
- with_filestem()
- add_extension()
- file_path()
Also change pop() to return a boolean instead of an owned copy of the
old filename.
Standardize the is_sep() functions to be the same in both posix and
windows, and re-export from path. Update extra::glob to use this.
Remove the usage of either, as it's going away.
Move the WindowsPath-specific methods out of WindowsPath and make them
top-level functions of path::windows instead. This way you cannot
accidentally write code that will fail to compile on non-windows
architectures without typing ::windows anywhere.
Remove GenericPath::from_c_str() and just impl BytesContainer for
CString instead.
Remove .join_path() and .push_path() and just implement BytesContainer
for Path instead.
Remove FilenameDisplay and add a boolean flag to Display instead.
Remove .each_parent(). It only had one caller, so just inline its
definition there.
Add a new trait BytesContainer that is implemented for both byte vectors
and strings.
Convert Path::from_vec and ::from_str to one function, Path::new().
Remove all the _str-suffixed mutation methods (push, join, with_*,
set_*) and modify the non-suffixed versions to use BytesContainer.
Remove the old path.
Rename path2 to path.
Update all clients for the new path.
Also make some miscellaneous changes to the Path APIs to help the
adoption process.
There's currently a fair amount of code which is being ignored on unnamed blocks
(which are the default now), and I opted to leave it commented out for now. I
intend on very soon revisiting on how we perform linking with extern crates in
an effort to support static linking.
For the benefit of the pretty printer we want to keep track of how
string literals in the ast were originally represented in the source
code.
This commit changes parser functions so they don't extract strings from
the token stream without at least also returning what style of string
literal it was. This is stored in the resulting ast node for string
literals, obviously, for the package id in `extern mod = r"package id"`
view items, for the inline asm in `asm!()` invocations.
For `asm!()`'s other arguments or for `extern "Rust" fn()` items, I just
the style of string, because it seemed disproportionally cumbersome to
thread that information through the string processing that happens with
those string literals, given the limited advantage raw string literals
would provide in these positions.
The other syntax extensions don't seem to store passed string literals
in the ast, so they also discard the style of strings they parse.
This commit is the culmination of my recent effort to refine Rust's notion of
privacy and visibility among crates. The major goals of this commit were to
remove privacy checking from resolve for the sake of sane error messages, and to
attempt a much more rigid and well-tested implementation of visibility
throughout rust. The implemented rules for name visibility are:
1. Everything pub from the root namespace is visible to anyone
2. You may access any private item of your ancestors.
"Accessing a private item" depends on what the item is, so for a function this
means that you can call it, but for a module it means that you can look inside
of it. Once you look inside a private module, any accessed item must be "pub
from the root" where the new root is the private module that you looked into.
These rules required some more analysis results to get propagated from trans to
privacy in the form of a few hash tables.
I added a new test in which my goal was to showcase all of the privacy nuances
of the language, and I hope to place any new bugs into this file to prevent
regressions.
Overall, I was unable to completely remove the notion of privacy from resolve.
One use of privacy is for dealing with glob imports. Essentially a glob import
can only import *public* items from the destination, and because this must be
done at namespace resolution time, resolve must maintain the notion of "what
items are public in a module". There are some sad approximations of privacy, but
I unfortunately can't see clear methods to extract them outside.
The other use case of privacy in resolve now is one that must stick around
regardless of glob imports. When dealing with privacy, checking a private path
needs to know "what the last private thing was" when looking at a path. Resolve
is the only compiler pass which knows the answer to this question, so it
maintains the answer on a per-path resolution basis (works similarly to the
def_map generated).
Closes#8215
It is simply defined as `f64` across every platform right now.
A use case hasn't been presented for a `float` type defined as the
highest precision floating point type implemented in hardware on the
platform. Performance-wise, using the smallest precision correct for the
use case greatly saves on cache space and allows for fitting more
numbers into SSE/AVX registers.
If there was a use case, this could be implemented as simply a type
alias or a struct thanks to `#[cfg(...)]`.
Closes#6592
The mailing list thread, for reference:
https://mail.mozilla.org/pipermail/rust-dev/2013-July/004632.html
This fixes private statics and functions from being usable cross-crates, along
with some bad privacy error messages. This is a reopening of #8365 with all the
privacy checks in privacy.rs instead of resolve.rs (where they should be
anyway).
These maps of exported items will hopefully get used for generating
documentation by rustdoc
Closes#8592
In #8185 cross-crate condition handlers were fixed by ensuring that globals
didn't start appearing in different crates with different addressed. An
unfortunate side effect of that pull request is that constants weren't inlined
across crates (uint::bits is unknown to everything but libstd).
This commit fixes this inlining by using the `available_eternally` linkage
provided by LLVM. It partially reverts #8185, and then adds support for this
linkage type. The main caveat is that not all statics could be inlined into
other crates. Before this patch, all statics were considered "inlineable items",
but an unfortunate side effect of how we deal with `&static` and `&[static]`
means that these two cases cannot be inlined across crates. The translation of
constants was modified to propogate this condition of whether a constant
should be considered inlineable into other crates.
Closes#9036
While they may have the same name within various scopes, this changes static
names to use path_pretty_name to append some hash information at the end of the
symbol. We're then guaranteed that each static has a unique NodeId, so this
NodeId is as the "hash" of the pretty name.
Closes#9188
While they may have the same name within various scopes, this changes static
names to use path_pretty_name to append some hash information at the end of the
symbol. We're then guaranteed that each static has a unique NodeId, so this
NodeId is as the "hash" of the pretty name.
Closes#9188
This is a series of patches to modernize option and result. The highlights are:
* rename `.unwrap_or_default(value)` and etc to `.unwrap_or(value)`
* add `.unwrap_or_default()` that uses the `Default` trait
* add `Default` implementations for vecs, HashMap, Option
* add `Option.and(T) -> Option<T>`, `Option.and_then(&fn() -> Option<T>) -> Option<T>`, `Option.or(T) -> Option<T>`, and `Option.or_else(&fn() -> Option<T>) -> Option<T>`
* add `option::ToOption`, `option::IntoOption`, `option::AsOption`, `result::ToResult`, `result::IntoResult`, `result::AsResult`, `either::ToEither`, and `either::IntoEither`, `either::AsEither`
* renamed `Option::chain*` and `Result::chain*` to `and_then` and `or_else` to avoid the eventual collision with `Iterator.chain`.
* Added a bunch of impls of `Default`
* Added a `#[deriving(Default)]` syntax extension
* Removed impls of `Zero` for `Option<T>` and vecs.
As per rustpkg.md, rustpkg now builds in a target-specific
subdirectory of build/, and installs libraries into a target-specific
subdirectory of lib.
Closes#8672
This removes another large chunk of this odd 'clownshoes' identifier showing up
in symbol names. These all originated from external crates because the encoded
items were encoded independently of the paths calculated in ast_map. The
encoding of these paths now uses the helper function in ast_map to calculate the
"pretty name" for an impl block.
Unfortunately there is still no information about generics in the symbol name,
but it's certainly vastly better than before
hash::__extensions__::write::_version::v0.8
becomes
hash::Writer$SipState::write::hversion::v0.8
This also fixes bugs in which lots of methods would show up as `meth_XXX`, they
now only show up as `meth` and throw some extra characters onto the version
string.
These commits fix bugs related to identically named statics in functions of implementations in various situations. The commit messages have most of the information about what bugs are being fixed and why.
As a bonus, while I was messing around with name mangling, I improved the backtraces we'll get in gdb by removing `__extensions__` for the trait/type being implemented and by adding the method name as well. Yay!
There are 6 new compiler recognised attributes: deprecated, experimental,
unstable, stable, frozen, locked (these levels are taken directly from
Node's "stability index"[1]). These indicate the stability of the
item to which they are attached; e.g. `#[deprecated] fn foo() { .. }`
says that `foo` is deprecated.
This comes with 3 lints for the first 3 levels (with matching names) that
will detect the use of items marked with them (the `unstable` lint
includes items with no stability attribute). The attributes can be given
a short text note that will be displayed by the lint. An example:
#[warn(unstable)]; // `allow` by default
#[deprecated="use `bar`"]
fn foo() { }
#[stable]
fn bar() { }
fn baz() { }
fn main() {
foo(); // "warning: use of deprecated item: use `bar`"
bar(); // all fine
baz(); // "warning: use of unmarked item"
}
The lints currently only check the "edges" of the AST: i.e. functions,
methods[2], structs and enum variants. Any stability attributes on modules,
enums, traits and impls are not checked.
[1]: http://nodejs.org/api/documentation.html
[2]: the method check is currently incorrect and doesn't work.
As with the previous commit, this is targeted at removing the possibility of
collisions between statics. The main use case here is when there's a
type-parametric function with an inner static that's compiled as a library.
Before this commit, any impl would generate a path item of "__extensions__".
This changes this identifier to be a "pretty name", which is either the last
element of the path of the trait implemented or the last element of the type's
path that's being implemented. That doesn't quite cut it though, so the (trait,
type) pair is hashed and again used to append information to the symbol.
Essentially, __extensions__ was removed for something nicer for debugging, and
then some more information was added to symbol name by including a hash of the
trait being implemented and type it's being implemented for. This should prevent
colliding names for inner statics in regular functions with similar names.
