Currently, rustc requires that a linkage be a product of 100% rlibs or 100%
dylibs. This is to satisfy the requirement that each object appear at most once
in the final output products. This is a bit limiting, and the upcoming libcore
library cannot exist as a dylib, so these rules must change.
The goal of this commit is to enable *some* use cases for mixing rlibs and
dylibs, primarily libcore's use case. It is not targeted at allowing an
exhaustive number of linkage flavors.
There is a new dependency_format module in rustc which calculates what format
each upstream library should be linked as in each output type of the current
unit of compilation. The module itself contains many gory details about what's
going on here.
cc #10729
This has long since not been too relevant since the introduction of many crate
type outputs. This commit removes the flag entirely, adjusting all logic to do
the most reasonable thing when building both a library and an executable.
Closes#13337
This has long since not been too relevant since the introduction of many crate
type outputs. This commit removes the flag entirely, adjusting all logic to do
the most reasonable thing when building both a library and an executable.
Closes#13337
Currently, rustc requires that a linkage be a product of 100% rlibs or 100%
dylibs. This is to satisfy the requirement that each object appear at most once
in the final output products. This is a bit limiting, and the upcoming libcore
library cannot exist as a dylib, so these rules must change.
The goal of this commit is to enable *some* use cases for mixing rlibs and
dylibs, primarily libcore's use case. It is not targeted at allowing an
exhaustive number of linkage flavors.
There is a new dependency_format module in rustc which calculates what format
each upstream library should be linked as in each output type of the current
unit of compilation. The module itself contains many gory details about what's
going on here.
cc #10729
It didn't work because it tried to call itself but symbols are not
exported as default in executables.
Note that `fun5` is not internal anymore since it is in library.
The filestem of the desired output isn't necessarily a valid crate id, and
calling unwrap() will trigger an ICE in rustc. This tries a little harder to
infer a "valid crate id" from a crate, with an eventual fallback to a generic
crate id if alll else fails.
Closes#11107
Before, the `--crate-file-name` flag only checked crate attributes for
possible crate types. Now, if any type is specified by one or more
`--crate-type` flags, only the filenames for those types will be
emitted, and any types specified by crate attributes will be ignored.
Before, normal compilation and the --crate-file-name flag would
generate output based on both #![crate_type] attributes and
--crate-type flags. Now, if one or more flag is specified by command
line, only those will be used.
Closes#11573.
This bug was introduced in #13384 by accident, and this commit continues the
work of #13384 by finishing support for loading a syntax extension crate without
registering it with the local cstore.
Closes#13495
This bug was introduced in #13384 by accident, and this commit continues the
work of #13384 by finishing support for loading a syntax extension crate without
registering it with the local cstore.
Closes#13495
Previously, upstream C libraries were linked in a nondeterministic fashion
because they were collected through iter_crate_data() which is a nodeterministic
traversal of a hash map. When upstream rlibs had interdependencies among their
native libraries (such as libfoo depending on libc), then the ordering would
occasionally be wrong, causing linkage to fail.
This uses the topologically sorted list of libraries to collect native
libraries, so if a native library depends on libc it just needs to make sure
that the rust crate depends on liblibc.
`Reader`, `Writer`, `MemReader`, `MemWriter`, and `MultiWriter` now work with `Vec<u8>` instead of `~[u8]`. This does introduce some extra copies since `from_utf8_owned` isn't usable anymore, but I think that can't be helped until `~str`'s representation changes.
Rust currently defaults to `RelocPIC` regardless. This patch adds a new
codegen option that allows choosing different relocation-model. The
available models are:
- default (Use the target-specific default model)
- static
- pic
- no-pic
For a more detailed information use `llc --help`
Rust currently defaults to `RelocPIC` regardless. This patch adds a new
codegen option that allows choosing different relocation-model. The
available models are:
- default (Use the target-specific default model)
- static
- pic
- no-pic
For a more detailed information use `llc --help`
Fix#13266.
There is a little bit of acrobatics in the definition of `crate_paths`
to avoid calling `clone()` on the dylib/rlib unless we actually are
going to need them.
The other oddity is that I have replaced the `root_ident: Option<&str>`
parameter with a `root: &Option<CratePaths>`, which may surprise one
who was expecting to see something like: `root: Option<&CratePaths>`.
I went with the approach here because I could not come up with code for
the alternative that was acceptable to the borrow checker.
All it checks, unfortunately, is that you actually printed at least
two lines for crateA paths and at least one line for crateB paths.
But that's enough to capture the spirit of the bug, I think. I did
not bother trying to verify that the paths themselves reflected where
the crates end up.
This is the final nail in the coffin for the crate map. The `start` function for
libgreen now has a new added parameter which is the event loop factory instead
of inferring it from the crate map. The two current valid values for this
parameter are `green::basic::event_loop` and `rustuv::event_loop`.
Fixes#12992
Store compressed bitcode files in rlibs with a different extension. Compression doesn't interfere with --emit=bc.
Regression test compares outputs.
The compiler will no longer inject libgreen as the default runtime for rust
programs, this commit switches it over to libnative by default. Now that
libnative has baked for some time, it is ready enough to start getting more
serious usage as the default runtime for rustc generated binaries.
We've found that there isn't really a correct decision in choosing a 1:1 or M:N
runtime as a default for all applications, but it seems that a larger number of
programs today would work more reasonable with a native default rather than a
green default.
With this commit come a number of bugfixes:
* The main native task is now named "<main>"
* The main native task has the stack bounds set up properly
* #[no_uv] was renamed to #[no_start]
* The core-run-destroy test was rewritten for both libnative and libgreen and
one of the tests was modified to be more robust.
* The process-detach test was locked to libgreen because it uses signal handling
Previously, any library of the pattern `lib<name>-<hash>-<version>.so` was
>considered a candidate (rightly so) for loading a crate. Sets are generated for
each unique `<hash>`, and then from these sets a candidate is selected. If a set
contained more than one element, then it immediately generated an error saying
that multiple copies of the same dylib were found.
This is incorrect because each candidate needs to be validated to actually
contain a rust library (valid metadata). This commit alters the logic to filter
each set of candidates for a hash to only libraries which are actually rust
libraries. This means that if multiple false positives are found with the right
name pattern, they're all ignored.
Closes#13010
This commit moves all logging out of the standard library into an external
crate. This crate is the new crate which is responsible for all logging macros
and logging implementation. A few reasons for this change are:
* The crate map has always been a bit of a code smell among rust programs. It
has difficulty being loaded on almost all platforms, and it's used almost
exclusively for logging and only logging. Removing the crate map is one of the
end goals of this movement.
* The compiler has a fair bit of special support for logging. It has the
__log_level() expression as well as generating a global word per module
specifying the log level. This is unfairly favoring the built-in logging
system, and is much better done purely in libraries instead of the compiler
itself.
* Initialization of logging is much easier to do if there is no reliance on a
magical crate map being available to set module log levels.
* If the logging library can be written outside of the standard library, there's
no reason that it shouldn't be. It's likely that we're not going to build the
highest quality logging library of all time, so third-party libraries should
be able to provide just as high-quality logging systems as the default one
provided in the rust distribution.
With a migration such as this, the change does not come for free. There are some
subtle changes in the behavior of liblog vs the previous logging macros:
* The core change of this migration is that there is no longer a physical
log-level per module. This concept is still emulated (it is quite useful), but
there is now only a global log level, not a local one. This global log level
is a reflection of the maximum of all log levels specified. The previously
generated logging code looked like:
if specified_level <= __module_log_level() {
println!(...)
}
The newly generated code looks like:
if specified_level <= ::log::LOG_LEVEL {
if ::log::module_enabled(module_path!()) {
println!(...)
}
}
Notably, the first layer of checking is still intended to be "super fast" in
that it's just a load of a global word and a compare. The second layer of
checking is executed to determine if the current module does indeed have
logging turned on.
This means that if any module has a debug log level turned on, all modules
with debug log levels get a little bit slower (they all do more expensive
dynamic checks to determine if they're turned on or not).
Semantically, this migration brings no change in this respect, but
runtime-wise, this will have a perf impact on some code.
* A `RUST_LOG=::help` directive will no longer print out a list of all modules
that can be logged. This is because the crate map will no longer specify the
log levels of all modules, so the list of modules is not known. Additionally,
warnings can no longer be provided if a malformed logging directive was
supplied.
The new "hello world" for logging looks like:
#[phase(syntax, link)]
extern crate log;
fn main() {
debug!("Hello, world!");
}
Linker argument order with respect to libraries is important enough that we
shouldn't be attempting to filter out libraries getting passed through to the
linker. When linking with a native library that has multiple dependant native
libraries, it's useful to have control over the link argument order.
When the metadata format changes, old libraries often cause librustc to abort
when reading their metadata. This should all change with the introduction of SVH
markers, but the loader for crates should gracefully handle libraries without
SVH markers still.
This commit adds support for tripping fewer assertions when loading libraries by
using maybe_get_doc when initially parsing metadata. It's still possible for
some libraries to fall through the cracks, but this should deal with a fairly
large number of them up front.
With linkers on unix systems, libraries on the right of the command line are
used to resolve symbols in those on the left of the command line. This means
that arguments must have a right-to-left dependency chain (things on the left
depend on things on the right).
This is currently done by ordering the linker arguments as
1. Local object
2. Local native libraries
3. Upstream rust libraries
4. Upstream native libraries
This commit swaps the order of 2 and 3 so upstream rust libraries have access to
local native libraries. It has been seen that some upstream crates don't specify
the library that they link to because the name varies per platform (e.g.
lua/glfw/etc).
This commit enables building these libraries by allowing the upstream rust crate
to have access to local native libraries. I believe that the failure mode for
this scheme is when an upstream rust crate depends on a symbol in an upstream
library which is then redefined in a local library. This failure mode is
incredibly uncommon, and the failure mode also varies per platform (OSX behaves
differently), so I believe that a change like this is fine to make.
Closes#12446
Apparently weak linkage and dlopen aren't quite working out for applications
like servo on android. There appears to be a bug or two in how android loads
dynamic libraries and for some reason libservo.so isn't being found.
As a temporary solution, add an extern "C" function to libstd which can be
called if you have a handle to the crate map manually. When crawling the crate
map, we then check this manual symbol before falling back to the old solutions.
cc #11731
They are still are not completely correct, since it does not handle
graphemes at all, just codepoints, but at least it handles the common
case correctly.
The calculation was previously very wrong (rather than just a little bit
wrong): it wasn't accounting for the fact that every character is 1
byte, and so multibyte characters were pretending to be zero width.
cc #8706
file.
Previously multibyte UTF-8 chars were being recorded as byte offsets
from the start of the file, and then later compared against global byte
positions, resulting in the compiler possibly thinking it had a byte
position pointing inside a multibyte character, if there were multibyte
characters in any non-crate files. (Although, sometimes the byte offsets
line up just right to not ICE, but that was a coincidence.)
Fixes#11136.
Fixes#11178.
This commit implements a layman's version of realpath() for metadata::loader to
use in order to not error on symlinks pointing to the same file.
Closes#12459
This adds simple syntax highlighting based off libsyntax's lexer to be sure to
stay up to date with rust's grammar. Some of the highlighting is a bit ad-hoc,
but it definitely seems to get the job done!
This currently doesn't highlight rustdoc-rendered function signatures and
structs that are emitted to each page because the colors already signify what's
clickable and I think we'd have to figure out a different scheme before
colorizing them. This does, however, colorize all code examples and source code.
Closes#11393
This adds simple syntax highlighting based off libsyntax's lexer to be sure to
stay up to date with rust's grammar. Some of the highlighting is a bit ad-hoc,
but it definitely seems to get the job done!
This currently doesn't highlight rustdoc-rendered function signatures and
structs that are emitted to each page because the colors already signify what's
clickable and I think we'd have to figure out a different scheme before
colorizing them. This does, however, colorize all code examples and source code.
Closes#11393
When creating a staticlib, it unzips all static archives it finds and then
inserts the files manually into the output file. This process is done through
`ar`, and `ar` doesn't like if you specify you want to add files and you don't
give it any files.
This case arose whenever you linked to an archive that didn't have any contents
or all of the contents were filtered out. This just involved ignoring the case
where the number of inputs we have is 0, because we don't have any files to add
anyway.
When creating a staticlib, it unzips all static archives it finds and then
inserts the files manually into the output file. This process is done through
`ar`, and `ar` doesn't like if you specify you want to add files and you don't
give it any files.
This case arose whenever you linked to an archive that didn't have any contents
or all of the contents were filtered out. This just involved ignoring the case
where the number of inputs we have is 0, because we don't have any files to add
anyway.
This commit removes the -c, --emit-llvm, -s, --rlib, --dylib, --staticlib,
--lib, and --bin flags from rustc, adding the following flags:
* --emit=[asm,ir,bc,obj,link]
* --crate-type=[dylib,rlib,staticlib,bin,lib]
The -o option has also been redefined to be used for *all* flavors of outputs.
This means that we no longer ignore it for libraries. The --out-dir remains the
same as before.
The new logic for files that rustc emits is as follows:
1. Output types are dictated by the --emit flag. The default value is
--emit=link, and this option can be passed multiple times and have all
options stacked on one another.
2. Crate types are dictated by the --crate-type flag and the #[crate_type]
attribute. The flags can be passed many times and stack with the crate
attribute.
3. If the -o flag is specified, and only one output type is specified, the
output will be emitted at this location. If more than one output type is
specified, then the filename of -o is ignored, and all output goes in the
directory that -o specifies. The -o option always ignores the --out-dir
option.
4. If the --out-dir flag is specified, all output goes in this directory.
5. If -o and --out-dir are both not present, all output goes in the current
directory of the process.
6. When multiple output types are specified, the filestem of all output is the
same as the name of the CrateId (derived from a crate attribute or from the
filestem of the crate file).
Closes#7791Closes#11056Closes#11667
Currently any line starting with `#` is filtered from the output,
including line like `#[deriving]`; this patch makes it so lines are only
filtered when followed by a space similar to the current behaviour of
the tutorial/manual tester.
* vec::raw::to_ptr is gone
* Pausible => Pausable
* Removing @
* Calling the main task "<main>"
* Removing unused imports
* Removing unused mut
* Bringing some libextra tests up to date
* Allowing compiletest to work at stage0
* Fixing the bootstrap-from-c rmake tests
* assert => rtassert in a few cases
* printing to stderr instead of stdout in fail!()
Right now the --crate-id and related flags are all process *after* the entire
crate is parsed. This is less than desirable when used with makefiles because it
means that just to learn the output name of the crate you have to parse the
entire crate (unnecessary).
This commit changes the behavior to lift the handling of these flags much sooner
in the compilation process. This allows us to not have to parse the entire crate
and only have to worry about parsing the crate attributes themselves. The
related methods have all been updated to take an array of attributes rather than
a crate.
Additionally, this ceases duplication of the "what output are we producing"
logic in order to correctly handle things in the case of --test.
Finally, this adds tests for all of this functionality to ensure that it does
not regress.
By performing this logic very late in the build process, it ended up leading to
bugs like those found in #10973 where certain stages of the build process
expected a particular output format which didn't end up being the case. In order
to fix this, the build output generation is moved very early in the build
process to the absolute first thing in phase 2.
Closes#10973
This replaces the link meta attributes with a pkgid attribute and uses a hash
of this as the crate hash. This makes the crate hash computable by things
other than the Rust compiler. It also switches the hash function ot SHA1 since
that is much more likely to be available in shell, Python, etc than SipHash.
Fixes#10188, #8523.
This commit implements LTO for rust leveraging LLVM's passes. What this means
is:
* When compiling an rlib, in addition to insdering foo.o into the archive, also
insert foo.bc (the LLVM bytecode) of the optimized module.
* When the compiler detects the -Z lto option, it will attempt to perform LTO on
a staticlib or binary output. The compiler will emit an error if a dylib or
rlib output is being generated.
* The actual act of performing LTO is as follows:
1. Force all upstream libraries to have an rlib version available.
2. Load the bytecode of each upstream library from the rlib.
3. Link all this bytecode into the current LLVM module (just using llvm
apis)
4. Run an internalization pass which internalizes all symbols except those
found reachable for the local crate of compilation.
5. Run the LLVM LTO pass manager over this entire module
6a. If assembling an archive, then add all upstream rlibs into the output
archive. This ignores all of the object/bitcode/metadata files rust
generated and placed inside the rlibs.
6b. If linking a binary, create copies of all upstream rlibs, remove the
rust-generated object-file, and then link everything as usual.
As I have explained in #10741, this process is excruciatingly slow, so this is
*not* turned on by default, and it is also why I have decided to hide it behind
a -Z flag for now. The good news is that the binary sizes are about as small as
they can be as a result of LTO, so it's definitely working.
Closes#10741Closes#10740
This is inspired by a mystifying linker failure when using `pkg-config` to
generate the linker args: `pkg-config` produces output that ends in a
space, thus resulting in an empty linker argument.
Also added some updates to the concerning error messages that helped
spotting this bug.
In #10422, I didn't actually test to make sure that the '-Z gen-crate-map'
option was usable before I implemented it. The crate map was indeed generated
when '-Z gen-crate-map' was specified, but the I/O factory slot was empty
because of an extra check in trans about filling in that location.
This commit both fixes that location, and checks in a "fancy test" which does
lots of fun stuff. The test will use the rustc library to compile a rust crate,
and then compile a C program to link against that crate and run the C program.
To my knowledge this is the first test of its kind, so it's a little ad-hoc, but
it seems to get the job done. We could perhaps generalize running tests like
this, but for now I think it's fine to have this sort of functionality tucked
away in a test.
This infrastructure is meant to support runnings tests that involve various
interesting interdependencies about the types of crates being linked or possibly
interacting with C libraries. The goal of these make tests is to not restrict
them to a particular test runner, but allow each test to run its own tests.
To this end, there is a new src/test/run-make directory which has sub-folders of
tests. Each test requires a `Makefile`, and running the tests constitues simply
running `make` inside the directory. The new target is `check-stageN-rmake`.
These tests will have the destination directory (as TMPDIR) and the local rust
compiler (as RUSTC) passed along to them. There is also some helpful
cross-platform utilities included in src/test/run-make/tools.mk to aid with
compiling C programs and running them.
The impetus for adding this new test suite is to allow various interesting forms
of testing rust linkage. All of the tests initially added are various flavors of
compiling Rust and C with one another as well as just making sure that rust
linkage works in general.
Closes#10434
