This commit implements LTO for rust leveraging LLVM's passes. What this means
is:
* When compiling an rlib, in addition to insdering foo.o into the archive, also
insert foo.bc (the LLVM bytecode) of the optimized module.
* When the compiler detects the -Z lto option, it will attempt to perform LTO on
a staticlib or binary output. The compiler will emit an error if a dylib or
rlib output is being generated.
* The actual act of performing LTO is as follows:
1. Force all upstream libraries to have an rlib version available.
2. Load the bytecode of each upstream library from the rlib.
3. Link all this bytecode into the current LLVM module (just using llvm
apis)
4. Run an internalization pass which internalizes all symbols except those
found reachable for the local crate of compilation.
5. Run the LLVM LTO pass manager over this entire module
6a. If assembling an archive, then add all upstream rlibs into the output
archive. This ignores all of the object/bitcode/metadata files rust
generated and placed inside the rlibs.
6b. If linking a binary, create copies of all upstream rlibs, remove the
rust-generated object-file, and then link everything as usual.
As I have explained in #10741, this process is excruciatingly slow, so this is
*not* turned on by default, and it is also why I have decided to hide it behind
a -Z flag for now. The good news is that the binary sizes are about as small as
they can be as a result of LTO, so it's definitely working.
Closes#10741Closes#10740
Right now whenever an rlib file is linked against, all of the metadata from the
rlib is pulled in to the final staticlib or binary. The reason for this is that
the metadata is currently stored in a section of the object file. Note that this
is intentional for dynamic libraries in order to distribute metadata bundled
with static libraries.
This commit alters the situation for rlib libraries to instead store the
metadata in a separate file in the archive. In doing so, when the archive is
passed to the linker, none of the metadata will get pulled into the result
executable. Furthermore, the metadata file is skipped when assembling rlibs into
an archive.
The snag in this implementation comes with multiple output formats. When
generating a dylib, the metadata needs to be in the object file, but when
generating an rlib this needs to be separate. In order to accomplish this, the
metadata variable is inserted into an entirely separate LLVM Module which is
then codegen'd into a different location (foo.metadata.o). This is then linked
into dynamic libraries and silently ignored for rlib files.
While changing how metadata is inserted into archives, I have also stopped
compressing metadata when inserted into rlib files. We have wanted to stop
compressing metadata, but the sections it creates in object file sections are
apparently too large. Thankfully if it's just an arbitrary file it doesn't
matter how large it is.
I have seen massive reductions in executable sizes, as well as staticlib output
sizes (to confirm that this is all working).
Last LLVM update seems to have fixed whatever prevented LLVM integrated assembler from generating correct unwind tables on Windows. This PR switches Windows builds to use internal assembler by default.
Compilation via external assembler can still be requested via the newly added `-Z no-integrated-as` option.
Closes#8809
LLVM's JIT has been updated numerous times, and we haven't been tracking it at
all. The existing LLVM glue code no longer compiles, and the JIT isn't used for
anything currently.
This also rebases out the FixedStackSegment support which we have added to LLVM.
None of this is still in use by the compiler, and there's no need to keep this
functionality around inside of LLVM.
This is needed to unblock #10708 (where we're tripping an LLVM assertion).
This reverts commit c54427ddfb.
Leave the #[ignores] in that were added to rustpkg tests.
Conflicts:
src/librustc/driver/driver.rs
src/librustc/metadata/creader.rs
This is inspired by a mystifying linker failure when using `pkg-config` to
generate the linker args: `pkg-config` produces output that ends in a
space, thus resulting in an empty linker argument.
Also added some updates to the concerning error messages that helped
spotting this bug.
This commit implements the support necessary for generating both intermediate
and result static rust libraries. This is an implementation of my thoughts in
https://mail.mozilla.org/pipermail/rust-dev/2013-November/006686.html.
When compiling a library, we still retain the "lib" option, although now there
are "rlib", "staticlib", and "dylib" as options for crate_type (and these are
stackable). The idea of "lib" is to generate the "compiler default" instead of
having too choose (although all are interchangeable). For now I have left the
"complier default" to be a dynamic library for size reasons.
Of the rust libraries, lib{std,extra,rustuv} will bootstrap with an
rlib/dylib pair, but lib{rustc,syntax,rustdoc,rustpkg} will only be built as a
dynamic object. I chose this for size reasons, but also because you're probably
not going to be embedding the rustc compiler anywhere any time soon.
Other than the options outlined above, there are a few defaults/preferences that
are now opinionated in the compiler:
* If both a .dylib and .rlib are found for a rust library, the compiler will
prefer the .rlib variant. This is overridable via the -Z prefer-dynamic option
* If generating a "lib", the compiler will generate a dynamic library. This is
overridable by explicitly saying what flavor you'd like (rlib, staticlib,
dylib).
* If no options are passed to the command line, and no crate_type is found in
the destination crate, then an executable is generated
With this change, you can successfully build a rust program with 0 dynamic
dependencies on rust libraries. There is still a dynamic dependency on
librustrt, but I plan on removing that in a subsequent commit.
This change includes no tests just yet. Our current testing
infrastructure/harnesses aren't very amenable to doing flavorful things with
linking, so I'm planning on adding a new mode of testing which I believe belongs
as a separate commit.
Closes#552
Now the privacy pass returns enough information that other passes do not need to duplicate the visibility rules, and the missing_doc implementation is more consistent with other lint checks.
Previously, the `exported_items` set created by the privacy pass was
incomplete. Specifically, it did not include items that had been defined
at a private path but then `pub use`d at a public path. This commit
finds all crate exports during the privacy pass. Consequently, some code
in the reachable pass and in rustdoc is no longer necessary. This commit
then removes the separate `MissingDocLintVisitor` lint pass, opting to
check missing_doc lint in the same pass as the other lint checkers using
the visibility result computed by the privacy pass.
Fixes#9777.
These two attributes are no longer useful now that Rust has decided to leave
segmented stacks behind. It is assumed that the rust task's stack is always
large enough to make an FFI call (due to the stack being very large).
There's always the case of stack overflow, however, to consider. This does not
change the behavior of stack overflow in Rust. This is still normally triggered
by the __morestack function and aborts the whole process.
C stack overflow will continue to corrupt the stack, however (as it did before
this commit as well). The future improvement of a guard page at the end of every
rust stack is still unimplemented and is intended to be the mechanism through
which we attempt to detect C stack overflow.
Closes#8822Closes#10155
This adds an other ABI option which allows a custom selection over the target
architecture and OS. The only current candidate for this change is that kernel32
on win32 uses stdcall, but on win64 it uses the cdecl calling convention.
Otherwise everywhere else this is defined as using the Cdecl calling convention.
cc #10049Closes#8774
This adds an other ABI option which allows a custom selection over the target
architecture and OS. The only current candidate for this change is that kernel32
on win32 uses stdcall, but on win64 it uses the cdecl calling convention.
Otherwise everywhere else this is defined as using the Cdecl calling convention.
cc #10049Closes#8774
This renames to_str_ascii to as_str_ascii and makes it non-copying,
which is possible now that strings no longer have a hidden extra
byte/null terminator.
Fixes#6120.
This fleshes out the io::file module a fair bit more, adding all of the functionality that I can think of that we would want. Some questions about the representation which I'm curious about:
* I modified `FileStat` to be a little less platform-agnostic, but it's still fairly platform-specific. I don't want to hide information that we have, but I don't want to depend on this information being available. One possible route is to have an `extra` field which has all this os-dependent stuff which is clearly documented as it should be avoided.
* Does it make sense for directory functions to be top-level functions instead of static methods? It seems silly to import `std::rt::io::file` and `std::rt::io::File` at the top of files that need to deal with directories and files.
This renames the `file` module to `fs` because that more accurately describes
its current purpose (manipulating the filesystem, not just files).
Additionally, this adds an UnstableFileStat structure as a nested structure of
FileStat to signify that the fields should not be depended on. The structure is
currently flagged with #[unstable], but it's unlikely that it has much meaning.
Closes#10241
This file did not respect the #[link(name = "...")] attribute when it was
clearly intended to do so. The problem is that the crate attributes just weren't
passed in. This causes lots of problems in rust today because the object file
for all our libraries is inferred to be 'lib.o' because all of the files are
called 'lib.rs'.
I tried to figure out a good way to test for this, but I wasn't able to come up
with a good way that fit into our current testing framework. Nonetheless, I have
tested this locally and object files get named as they should. This should fix
compiling with `make -jN` again (because the object files are all different
again).
This adds bindings to the remaining functions provided by libuv, all of which
are useful operations on files which need to get exposed somehow.
Some highlights:
* Dropped `FileReader` and `FileWriter` and `FileStream` for one `File` type
* Moved all file-related methods to be static methods under `File`
* All directory related methods are still top-level functions
* Created `io::FilePermission` types (backed by u32) that are what you'd expect
* Created `io::FileType` and refactored `FileStat` to use FileType and
FilePermission
* Removed the expanding matrix of `FileMode` operations. The mode of reading a
file will not have the O_CREAT flag, but a write mode will always have the
O_CREAT flag.
Closes#10130Closes#10131Closes#10121
This commit moves all thread-blocking I/O functions from the std::os module.
Their replacements can be found in either std::rt::io::file or in a hidden
"old_os" module inside of native::file. I didn't want to outright delete these
functions because they have a lot of special casing learned over time for each
OS/platform, and I imagine that these will someday get integrated into a
blocking implementation of IoFactory. For now, they're moved to a private module
to prevent bitrot and still have tests to ensure that they work.
I've also expanded the extensions to a few more methods defined on Path, most of
which were previously defined in std::os but now have non-thread-blocking
implementations as part of using the current IoFactory.
The api of io::file is in flux, but I plan on changing it in the next commit as
well.
Closes#10057
Add a new trait BytesContainer that is implemented for both byte vectors
and strings.
Convert Path::from_vec and ::from_str to one function, Path::new().
Remove all the _str-suffixed mutation methods (push, join, with_*,
set_*) and modify the non-suffixed versions to use BytesContainer.
Remove the old path.
Rename path2 to path.
Update all clients for the new path.
Also make some miscellaneous changes to the Path APIs to help the
adoption process.
This fixes a bug in which the visibility rules were approximated by
reachability, but forgot to cover the case where a 'pub use' reexports a private
item. This fixes the commit by instead using the results of the privacy pass of
the compiler to create the initial working set of the reachability pass.
This may have the side effect of increasing the size of metadata, but it's
difficult to avoid for correctness purposes sadly.
Closes#9790
This commit is the culmination of my recent effort to refine Rust's notion of
privacy and visibility among crates. The major goals of this commit were to
remove privacy checking from resolve for the sake of sane error messages, and to
attempt a much more rigid and well-tested implementation of visibility
throughout rust. The implemented rules for name visibility are:
1. Everything pub from the root namespace is visible to anyone
2. You may access any private item of your ancestors.
"Accessing a private item" depends on what the item is, so for a function this
means that you can call it, but for a module it means that you can look inside
of it. Once you look inside a private module, any accessed item must be "pub
from the root" where the new root is the private module that you looked into.
These rules required some more analysis results to get propagated from trans to
privacy in the form of a few hash tables.
I added a new test in which my goal was to showcase all of the privacy nuances
of the language, and I hope to place any new bugs into this file to prevent
regressions.
Overall, I was unable to completely remove the notion of privacy from resolve.
One use of privacy is for dealing with glob imports. Essentially a glob import
can only import *public* items from the destination, and because this must be
done at namespace resolution time, resolve must maintain the notion of "what
items are public in a module". There are some sad approximations of privacy, but
I unfortunately can't see clear methods to extract them outside.
The other use case of privacy in resolve now is one that must stick around
regardless of glob imports. When dealing with privacy, checking a private path
needs to know "what the last private thing was" when looking at a path. Resolve
is the only compiler pass which knows the answer to this question, so it
maintains the answer on a per-path resolution basis (works similarly to the
def_map generated).
Closes#8215
A few features are now hidden behind various #[feature(...)] directives. These
include struct-like enum variants, glob imports, and macro_rules! invocations.
Closes#9304Closes#9305Closes#9306Closes#9331
Avoid allocating extra copies of strings by using "" instead of ~"" for
the debug options list and for the `time` function. This is a small
change, but it is in a path that's always executed.
It is simply defined as `f64` across every platform right now.
A use case hasn't been presented for a `float` type defined as the
highest precision floating point type implemented in hardware on the
platform. Performance-wise, using the smallest precision correct for the
use case greatly saves on cache space and allows for fitting more
numbers into SSE/AVX registers.
If there was a use case, this could be implemented as simply a type
alias or a struct thanks to `#[cfg(...)]`.
Closes#6592
The mailing list thread, for reference:
https://mail.mozilla.org/pipermail/rust-dev/2013-July/004632.html
This is broken, and results in poor performance due to the undefined
behaviour in the LLVM IR. LLVM's `mergefunc` is a *much* better way of
doing this since it merges based on the equality of the bytecode.
For example, consider `std::repr`. It generates different code per
type, but is not included in the type bounds of generics.
The `mergefunc` pass works for most of our code but currently hits an
assert on libstd. It is receiving attention upstream so it will be
ready soon, but I don't think removing this broken code should wait any
longer. I've opened #9536 about enabling it by default.
Closes#8651Closes#3547Closes#2537Closes#6971Closes#9222
This fixes private statics and functions from being usable cross-crates, along
with some bad privacy error messages. This is a reopening of #8365 with all the
privacy checks in privacy.rs instead of resolve.rs (where they should be
anyway).
These maps of exported items will hopefully get used for generating
documentation by rustdoc
Closes#8592
