Add a link validator to rustbuild
This commit was originally targeted at just adding a link checking script to the rustbuild system. This ended up snowballing a bit to extend rustbuild to be amenable to various tools we have as part of the build system in general.
There's a new `src/tools` directory which has a number of scripts/programs that are purely intended to be used as part of the build system and CI of this repository. This is currently inhabited by rustbook, the error index generator, and a new linkchecker script added as part of this PR. I suspect that more tools like compiletest, tidy scripts, snapshot scripts, etc will migrate their way into this directory over time.
The commit which adds the error index generator shows the steps necessary to add new tools to the build system, namely:
1. New steps are defined for building the tool and running the tool
2. The dependencies are configured
3. The steps are implemented
In terms of the link checker, these commits do a few things:
* A new `src/tools/linkchecker` script is added. This will read an entire documentation tree looking for broken relative links (HTTP links aren't followed yet).
* A large number of broken links throughout the documentation were fixed. Many of these were just broken when viewed from core as opposed to std, but were easily fixed.
* A few rustdoc bugs here and there were fixed
mk: Distribute fewer TARGET_CRATES
Right now everything in TARGET_CRATES is built by default for all non-fulldeps
tests and is distributed by default for all target standard library packages.
Currenly this includes a number of unstable crates which are rarely used such as
`graphviz` and `rbml`>
This commit trims down the set of `TARGET_CRATES`, moves a number of tests to
`*-fulldeps` as a result, and trims down the dependencies of libtest so we can
distribute fewer crates in the `rust-std` packages.
Right now everything in TARGET_CRATES is built by default for all non-fulldeps
tests and is distributed by default for all target standard library packages.
Currenly this includes a number of unstable crates which are rarely used such as
`graphviz` and `rbml`>
This commit trims down the set of `TARGET_CRATES`, moves a number of tests to
`*-fulldeps` as a result, and trims down the dependencies of libtest so we can
distribute fewer crates in the `rust-std` packages.
The `--disable-jemalloc` configure option has a failure mode where it will
create a distribution that is not compatible with other compilers. For example
the nightly for Linux will assume that it will link to jemalloc by default as
an allocator for executable crates. If, however, a standard library is used
which was built via `./configure --disable-jemalloc` then this will fail
because the jemalloc crate wasn't built.
While this seems somewhat reasonable as a niche situation, the same mechanism is
used for disabling jemalloc for platforms that just don't support it. For
example if the rumprun target is compiled then the sibiling Linux target *also*
doesn't have jemalloc. This is currently a problem for our cross-build nightlies
which build many targets. If rumprun is also built, it will disable jemalloc for
all targets, which isn't desired.
This commit moves the platform-specific disabling of jemalloc as hardcoded logic
into the makefiles that is scoped per-platform. This way when configuring
multiple targets **without the `--disable-jemalloc` option specified** all
targets will get jemalloc as they should.
This is because the tool compiler passes the name of the tool
as a command line `--cfg`. The improved session config parser
is stricter and no longer permits invalid meta items (such as
"error-index-generator").
When building with Cargo we need to detect `feature = "jemalloc"` to enable
jemalloc, so propagate this same change to the build system to pass the right
`--cfg` argument.
This aligns with unicode recommendations and should be stable for all future
unicode releases. See http://unicode.org/reports/tr31/#R3.
This renames `libsyntax::lexer::is_whitespace` to `is_pattern_whitespace`
so potentially breaks users of libsyntax.
this makes sure the checks run before typeck (which might use the constant or const
function to calculate an array length) and gives prettier error messages in case of for
loops and such (since they aren't expanded yet).
Use arena allocation instead of reference counting for `Module`s to fix memory leaks from `Rc` cycles.
A module references its module children and its import resolutions, and an import resolution references the module defining the imported name, so there is a cycle whenever a module imports something from an ancestor module.
For example,
```rust
mod foo { // `foo` references `bar`.
fn baz() {}
mod bar { // `bar` references the import.
use foo::baz; // The import references `foo`.
}
}
```
With this commit, metadata encoding and decoding can make use of
thread-local encoding and decoding contexts. These allow implementers
of serialize::Encodable and Decodable to access information and
datastructures that would otherwise not be available to them. For
example, we can automatically translate def-id and span information
during decoding because the decoding context knows which crate the
data is decoded from. Or it allows to make ty::Ty decodable because
the context has access to the ty::ctxt that is needed for creating
ty::Ty instances.
* Delete `sys::unix::{c, sync}` as these are now all folded into libc itself
* Update all references to use `libc` as a result.
* Update all references to the new flat namespace.
* Moves all windows bindings into sys::c
This commit is an implementation of [RFC 1183][rfc] which allows swapping out
the default allocator on nightly Rust. No new stable surface area should be
added as a part of this commit.
[rfc]: https://github.com/rust-lang/rfcs/pull/1183
Two new attributes have been added to the compiler:
* `#![needs_allocator]` - this is used by liballoc (and likely only liballoc) to
indicate that it requires an allocator crate to be in scope.
* `#![allocator]` - this is a indicator that the crate is an allocator which can
satisfy the `needs_allocator` attribute above.
The ABI of the allocator crate is defined to be a set of symbols that implement
the standard Rust allocation/deallocation functions. The symbols are not
currently checked for exhaustiveness or typechecked. There are also a number of
restrictions on these crates:
* An allocator crate cannot transitively depend on a crate that is flagged as
needing an allocator (e.g. allocator crates can't depend on liballoc).
* There can only be one explicitly linked allocator in a final image.
* If no allocator is explicitly requested one will be injected on behalf of the
compiler. Binaries and Rust dylibs will use jemalloc by default where
available and staticlibs/other dylibs will use the system allocator by
default.
Two allocators are provided by the distribution by default, `alloc_system` and
`alloc_jemalloc` which operate as advertised.
Closes#27389
This commit removes all morestack support from the compiler which entails:
* Segmented stacks are no longer emitted in codegen.
* We no longer build or distribute libmorestack.a
* The `stack_exhausted` lang item is no longer required
The only current use of the segmented stack support in LLVM is to detect stack
overflow. This is no longer really required, however, because we already have
guard pages for all threads and registered signal handlers watching for a
segfault on those pages (to print out a stack overflow message). Additionally,
major platforms (aka Windows) already don't use morestack.
This means that Rust is by default less likely to catch stack overflows because
if a function takes up more than one page of stack space it won't hit the guard
page. This is what the purpose of morestack was (to catch this case), but it's
better served with stack probes which have more cross platform support and no
runtime support necessary. Until LLVM supports this for all platform it looks
like morestack isn't really buying us much.
cc #16012 (still need stack probes)
Closes#26458 (a drive-by fix to help diagnostics on stack overflow)
The path we pass to rustc will be visible in panic messages and
backtraces: they will be user visible!
Avoid junk in these paths by passing relative paths to rustc.
For most advanced users, `libcore` or `libstd` in the path will be
a clue to the location -- inside our code, not theirs.
Store both the relative path to the source as well as the absolute.
Use the relative path where it matters, compiling the main crates,
instead of changing all of the build process to cope with relative
paths.
Example output after this patch:
```
$ ./testunwrap
thread '<main>' panicked at 'called `Option::unwrap()` on a `None` value', ../src/libcore/option.rs:362
$ RUST_BACKTRACE=1 ./testunwrap
thread '<main>' panicked at 'called `Option::unwrap()` on a `None` value', ../src/libcore/option.rs:362
stack backtrace:
1: 0x7ff59c1e9956 - sys::backtrace::write::h67a542fd2b201576des
at ../src/libstd/sys/unix/backtrace.rs:158
2: 0x7ff59c1ed5b6 - panicking::on_panic::h3d21c41cdd5c12d41Xw
at ../src/libstd/panicking.rs:58
3: 0x7ff59c1e7b6e - rt::unwind::begin_unwind_inner::h9f3a5440cebb8baeLDw
at ../src/libstd/rt/unwind/mod.rs:273
4: 0x7ff59c1e7f84 - rt::unwind::begin_unwind_fmt::h4fe8a903e0c296b0RCw
at ../src/libstd/rt/unwind/mod.rs:212
5: 0x7ff59c1eced7 - rust_begin_unwind
6: 0x7ff59c22c11a - panicking::panic_fmt::h00b0cd49c98a9220i5B
at ../src/libcore/panicking.rs:64
7: 0x7ff59c22b9e0 - panicking::panic::hf549420c0ee03339P3B
at ../src/libcore/panicking.rs:45
8: 0x7ff59c1e621d - option::Option<T>::unwrap::h501963526474862829
9: 0x7ff59c1e61b1 - main::hb5c91ce92347d1e6eaa
10: 0x7ff59c1f1c18 - rust_try_inner
11: 0x7ff59c1f1c05 - rust_try
12: 0x7ff59c1ef374 - rt::lang_start::h7e51e19c6677cffe5Sw
at ../src/libstd/rt/unwind/mod.rs:147
at ../src/libstd/rt/unwind/mod.rs:130
at ../src/libstd/rt/mod.rs:128
13: 0x7ff59c1e628e - main
14: 0x7ff59b3f6b44 - __libc_start_main
15: 0x7ff59c1e6078 - <unknown>
16: 0x0 - <unknown>
```
This commit adds a ./configure option called `--disable-elf-tls` which disables
ELF based TLS (that which is communicated to LLVM) on platforms which already
support it. OSX 10.6 does not support this form of TLS, and some users of Rust
need to target 10.6 and are unable to do so due to the usage of TLS. The
standard library will continue to use ELF based TLS on OSX by default (as the
officially supported platform is 10.7+), but this adds an option to compile the
standard library in a way that is compatible with 10.6.
Previously libmorestack.a and libcompiler-rt.a were installed, but link.exe
looks for morestack.lib and compiler-rt.lib by default, so we need to install
these with the correct name
This commit adds support to the makefiles, configuration script, and build
system to understand MUSL. This is broken up into a few parts:
* Any target of the form `*-musl` requires the `--musl-root` option to
`./configure` which will indicate the root of the MUSL installation. It is
also expected that there is a libunwind build inside of that installation
built against that MUSL.
* Objects from MUSL are copied into the build tree for Rust to be statically
linked into the appropriate Rust library.
* Objects for binary startup and shutdown are included in each Rust installation
by default for MUSL. This requires MUSL to only be installed on the machine
compiling rust. Only a linker will be necessary for compiling against MUSL on
a target machine.
Eventually a MUSL and/or libunwind build may be integrated by default into the
build but for now they are just always assumed to exist externally.
This patch
1. renames libunicode to librustc_unicode,
2. deprecates several pieces of libunicode (see below), and
3. removes references to deprecated functions from
librustc_driver and libsyntax. This may change pretty-printed
output from these modules in cases involving wide or combining
characters used in filenames, identifiers, etc.
The following functions are marked deprecated:
1. char.width() and str.width():
--> use unicode-width crate
2. str.graphemes() and str.grapheme_indices():
--> use unicode-segmentation crate
3. str.nfd_chars(), str.nfkd_chars(), str.nfc_chars(), str.nfkc_chars(),
char.compose(), char.decompose_canonical(), char.decompose_compatible(),
char.canonical_combining_class():
--> use unicode-normalization crate
This patch
1. renames libunicode to librustc_unicode,
2. deprecates several pieces of libunicode (see below), and
3. removes references to deprecated functions from
librustc_driver and libsyntax. This may change pretty-printed
output from these modules in cases involving wide or combining
characters used in filenames, identifiers, etc.
The following functions are marked deprecated:
1. char.width() and str.width():
--> use unicode-width crate
2. str.graphemes() and str.grapheme_indices():
--> use unicode-segmentation crate
3. str.nfd_chars(), str.nfkd_chars(), str.nfc_chars(), str.nfkc_chars(),
char.compose(), char.decompose_canonical(), char.decompose_compatible(),
char.canonical_combining_class():
--> use unicode-normalization crate
This commit ceases documentation-by-default of crates such as `term`,
`serialize`, and `alloc`. Crates like `term` and `rand` have duplicates on
`crates.io` and the search index entries generated in the local tree end up
only leading to confusion. Crates like the entire compiler infrastructure,
`flate`, or `rbml` don't need to be documented in such a prominent location.
This change also means that doc tests will no longer be run for crates beyond
the facade (e.g. `serialize` or `term`), but there were very few doc tests in
there to begin with.
Closes#22168
This saves a bunch of a time and will make distributions smaller, as well as
avoiding filling the implementors page with internal garbage. Turn it back on
with `--enable-compiler-docs` if you want compiler docs during development.
Crates behind the facade are only documented on nightly/dev builds (where they
can be used).
[breaking-change]
Closes#23772Closes#21297
This pulls out the implementations of most built-in lints into a
separate crate, to reduce edit-compile-test iteration times with
librustc_lint and increase parallelism. This should enable lints to be
refactored, added and deleted much more easily as it slashes the
edit-compile cycle to get a minimal working compiler to test with (`make
rustc-stage1`) from
librustc -> librustc_typeck -> ... -> librustc_driver ->
libcore -> ... -> libstd
to
librustc_lint -> librustc_driver -> libcore -> ... libstd
which is significantly faster, mainly due to avoiding the librustc build
itself.
The intention would be to move as much as possible of the infrastructure
into the crate too, but the plumbing is deeply intertwined with librustc
itself at the moment. Also, there are lints for which diagnostics are
registered directly in the compiler code, not in their own crate
traversal, and their definitions have to remain in librustc.
This is a [breaking-change] for direct users of the compiler APIs:
callers of `rustc::session::build_session` or
`rustc::session::build_session_` need to manually call
`rustc_lint::register_builtins` on their return value.
This should make #22206 easier.
The regex library was largely used for non-critical aspects of the compiler and
various external tooling. The library at this point is duplicated with its
out-of-tree counterpart and as such imposes a bit of a maintenance overhead as
well as compile time hit for the compiler itself.
The last major user of the regex library is the libtest library, using regexes
for filters when running tests. This removal means that the filtering has gone
back to substring matching rather than using regexes.
In accordance with [collections reform part 2][rfc] this macro has been moved to
an external [bitflags crate][crate] which is [available though
crates.io][cratesio]. Inside the standard distribution the macro has been moved
to a crate called `rustc_bitflags` for current users to continue using.
[rfc]: https://github.com/rust-lang/rfcs/blob/master/text/0509-collections-reform-part-2.md
[crate]: https://github.com/rust-lang/bitflags
[cratesio]: http://crates.io/crates/bitflags
The major user of `bitflags!` in terms of a public-facing possibly-stable API
today is the `FilePermissions` structure inside of `std::io`. This user,
however, will likely no longer use `bitflags!` after I/O reform has landed. To
prevent breaking APIs today, this structure remains as-is.
Current users of the `bitflags!` macro should add this to their `Cargo.toml`:
bitflags = "0.1"
and this to their crate root:
#[macro_use] extern crate bitflags;
Due to the removal of a public macro, this is a:
[breaking-change]
This pulls all of our long-form documentation into a single document,
nicknamed "the book" and formally titled "The Rust Programming
Language."
A few things motivated this change:
* People knew of The Guide, but not the individual Guides. This merges
them together, helping discoverability.
* You can get all of Rust's longform documentation in one place, which
is nice.
* We now have rustbook in-tree, which can generate this kind of
documentation. While its style is basic, the general idea is much
better: a table of contents on the left-hand side.
* Rather than a almost 10,000-line guide.md, there are now smaller files
per section.
This removes a large array of deprecated functionality, regardless of how
recently it was deprecated. The purpose of this commit is to clean out the
standard libraries and compiler for the upcoming alpha release.
Some notable compiler changes were to enable warnings for all now-deprecated
command line arguments (previously the deprecated versions were silently
accepted) as well as removing deriving(Zero) entirely (the trait was removed).
The distribution no longer contains the libtime or libregex_macros crates. Both
of these have been deprecated for some time and are available externally.
