rustc: Implement custom derive (macros 1.1)
This commit is an implementation of [RFC 1681] which adds support to the
compiler for first-class user-define custom `#[derive]` modes with a far more
stable API than plugins have today.
[RFC 1681]: https://github.com/rust-lang/rfcs/blob/master/text/1681-macros-1.1.md
The main features added by this commit are:
* A new `rustc-macro` crate-type. This crate type represents one which will
provide custom `derive` implementations and perhaps eventually flower into the
implementation of macros 2.0 as well.
* A new `rustc_macro` crate in the standard distribution. This crate will
provide the runtime interface between macro crates and the compiler. The API
here is particularly conservative right now but has quite a bit of room to
expand into any manner of APIs required by macro authors.
* The ability to load new derive modes through the `#[macro_use]` annotations on
other crates.
All support added here is gated behind the `rustc_macro` feature gate, both for
the library support (the `rustc_macro` crate) as well as the language features.
There are a few minor differences from the implementation outlined in the RFC,
such as the `rustc_macro` crate being available as a dylib and all symbols are
`dlsym`'d directly instead of having a shim compiled. These should only affect
the implementation, however, not the public interface.
This commit also ended up touching a lot of code related to `#[derive]`, making
a few notable changes:
* Recognized derive attributes are no longer desugared to `derive_Foo`. Wasn't
sure how to keep this behavior and *not* expose it to custom derive.
* Derive attributes no longer have access to unstable features by default, they
have to opt in on a granular level.
* The `derive(Copy,Clone)` optimization is now done through another "obscure
attribute" which is just intended to ferry along in the compiler that such an
optimization is possible. The `derive(PartialEq,Eq)` optimization was also
updated to do something similar.
---
One part of this PR which needs to be improved before stabilizing are the errors
and exact interfaces here. The error messages are relatively poor quality and
there are surprising spects of this such as `#[derive(PartialEq, Eq, MyTrait)]`
not working by default. The custom attributes added by the compiler end up
becoming unstable again when going through a custom impl.
Hopefully though this is enough to start allowing experimentation on crates.io!
test: Add a min-llvm-version directive
We've got tests which require a particular version of LLVM to run as they're
testing bug fixes. Our build system, however, supports multiple LLVM versions,
so we can't run these tests on all LLVM versions.
This adds a new `min-llvm-version` directive for tests so they can opt out of
being run on older versions of LLVM. This then namely applies that logic to the
`issue-36023.rs` test case and...
Closes#36138
This commit is an implementation of [RFC 1681] which adds support to the
compiler for first-class user-define custom `#[derive]` modes with a far more
stable API than plugins have today.
[RFC 1681]: https://github.com/rust-lang/rfcs/blob/master/text/1681-macros-1.1.md
The main features added by this commit are:
* A new `rustc-macro` crate-type. This crate type represents one which will
provide custom `derive` implementations and perhaps eventually flower into the
implementation of macros 2.0 as well.
* A new `rustc_macro` crate in the standard distribution. This crate will
provide the runtime interface between macro crates and the compiler. The API
here is particularly conservative right now but has quite a bit of room to
expand into any manner of APIs required by macro authors.
* The ability to load new derive modes through the `#[macro_use]` annotations on
other crates.
All support added here is gated behind the `rustc_macro` feature gate, both for
the library support (the `rustc_macro` crate) as well as the language features.
There are a few minor differences from the implementation outlined in the RFC,
such as the `rustc_macro` crate being available as a dylib and all symbols are
`dlsym`'d directly instead of having a shim compiled. These should only affect
the implementation, however, not the public interface.
This commit also ended up touching a lot of code related to `#[derive]`, making
a few notable changes:
* Recognized derive attributes are no longer desugared to `derive_Foo`. Wasn't
sure how to keep this behavior and *not* expose it to custom derive.
* Derive attributes no longer have access to unstable features by default, they
have to opt in on a granular level.
* The `derive(Copy,Clone)` optimization is now done through another "obscure
attribute" which is just intended to ferry along in the compiler that such an
optimization is possible. The `derive(PartialEq,Eq)` optimization was also
updated to do something similar.
---
One part of this PR which needs to be improved before stabilizing are the errors
and exact interfaces here. The error messages are relatively poor quality and
there are surprising spects of this such as `#[derive(PartialEq, Eq, MyTrait)]`
not working by default. The custom attributes added by the compiler end up
becoming unstable again when going through a custom impl.
Hopefully though this is enough to start allowing experimentation on crates.io!
syntax-[breaking-change]
add mips64-gnu and mips64el-gnu targets
With this commit one can build no_core (and probably no_std as well)
Rust programs for these targets. It's not yet possible to cross compile
std for these targets because rust-lang/libc doesn't know about the
mips64 architecture.
These targets have been tested by cross compiling the "smallest hello"
program (see code below) and then running it under QEMU.
``` rust
extern {
fn puts(_: *const u8);
}
fn start(_: isize, _: *const *const u8) -> isize {
unsafe {
let msg = b"Hello, world!\0";
puts(msg as *const _ as *const u8);
}
0
}
trait Copy {}
trait Sized {}
```
cc #36015
r? @alexcrichton
cc @brson
The cabi stuff is likely wrong. I just copied cabi_mips source and changed some `4`s to `8`s and `32`s to `64`s. It was enough to get libc's `puts` to work but I'd like someone familiar with this module to check it.
We've got tests which require a particular version of LLVM to run as they're
testing bug fixes. Our build system, however, supports multiple LLVM versions,
so we can't run these tests on all LLVM versions.
This adds a new `min-llvm-version` directive for tests so they can opt out of
being run on older versions of LLVM. This then namely applies that logic to the
`issue-36023.rs` test case and...
Closes#36138
Implement synchronization scheme for incr. comp. directory
This PR implements a copy-on-write-based synchronization scheme for the incremental compilation cache directory. For technical details, see the documentation at the beginning of `rustc_incremental/persist/fs.rs`.
The PR contains unit tests for some functions but for testing whether the scheme properly handles races, a more elaborate test setup would be needed. It would probably involve a small tool that allows to manipulate the incremental compilation directory in a controlled way and then letting a compiler instance run against directories in different states. I don't know if it's worth the trouble of adding another test category to `compiletest`, but I'd be happy to do so.
Fixes#32754Fixes#34957
initial support for s390x
A new target, `s390x-unknown-linux-gnu`, has been added to the compiler
and can be used to build no_core/no_std Rust programs.
Known limitations:
- librustc_trans/cabi_s390x.rs is missing. This means no support for
`extern "C" fn`.
- No support for this arch in libc. This means std can't be cross
compiled for this target.
r? @alexcrichton
This time I couldn't test running a binary cross compiled to this target under QEMU because the qemu-s390x that ships with Ubuntu 16.04 SIGABRTs with every s390x binary I run it with.
Change in binary size of `librustc_llvm.so`:
Without this commit (stage1): 41895736 bytes
With this commit (stage1): 42899016 bytes
~2.4% increase
A new target, `s390x-unknown-linux-gnu`, has been added to the compiler
and can be used to build no_core/no_std Rust programs.
Known limitations:
- librustc_trans/cabi_s390x.rs is missing. This means no support for
`extern "C" fn`.
- No support for this arch in libc. This means std can be cross compiled
for this target.
With this commit one can build no_core (and probably no_std as well)
Rust programs for these targets. It's not yet possible to cross compile
std for these targets because rust-lang/libc doesn't know about the
mips64 architecture.
These targets have been tested by cross compiling the "smallest hello"
program (see code below) and then running it under QEMU.
``` rust
#![feature(start)]
#![feature(lang_items)]
#![feature(no_core)]
#![no_core]
#[link(name = "c")]
extern {
fn puts(_: *const u8);
}
#[start]
fn start(_: isize, _: *const *const u8) -> isize {
unsafe {
let msg = b"Hello, world!\0";
puts(msg as *const _ as *const u8);
}
0
}
#[lang = "copy"]
trait Copy {}
#[lang = "sized"]
trait Sized {}
```
add mips-uclibc targets
These targets cover OpenWRT 15.05 devices, which use the soft float ABI
and the uclibc library. None of the other built-in mips targets covered
those devices (mips-gnu is hard float and glibc-based, mips-musl is
musl-based).
With this commit one can now build std for these devices using these
commands:
```
$ configure --enable-rustbuild --target=mips-unknown-linux-uclibc
$ make
```
cc #35673
---
r? @alexcrichton
cc @felixalias This is the target the rust-tessel project should be using.
Note that the libc crate doesn't support the uclibc library and will have to be updated. We are lucky that uclibc and glibc are somewhat similar and one can build std and even run the libc-test test suite with the current, unmodified libc. About that last part, I tried to run the libc-test and got a bunch of compile errors. I don't intend to fix them but I'll post some instruction about how to run libc-test in the rust-lang/libc issue tracker.
Kicking off libproc_macro
This PR introduces `libproc_macro`, which is currently quite bare-bones (just a few macro construction tools and an initial `quote!` macro).
This PR also introduces a few test cases for it, and an additional `shim` file (at `src/libsyntax/ext/proc_macro_shim.rs` to allow a facsimile usage of Macros 2.0 *today*!
These targets cover OpenWRT 15.05 devices, which use the soft float ABI
and the uclibc library. None of the other built-in mips targets covered
those devices (mips-gnu is hard float and glibc-based, mips-musl is
musl-based).
With this commit one can now build std for these devices using these
commands:
```
$ configure --enable-rustbuild --target=mips-unknown-linux-uclibc
$ make
```
cc #35673
add GNU make files for arm-unknown-linux-musleabi
For Yocto (Embedded Linux meta distro) Rust is provided via the [meta-rust layer](https://github.com/meta-rust/meta-rust). In this project there have been patches to add `arm-unknown-linux-musleabi`. Rust recently acquired that support via #35060 but only for rustbuild. meta-rust is currently only able to build Rust support with the existing GNU Makefiles. This adds `arm-unknown-linux-musleabi` support to Rust for the GNU Makefiles until meta-rust is able to sort out why using rustbuild does not work for it.
/cc @srwalter @derekstraka @jmesmon @japaric
add -mrelax-relocations=no to i686-musl and i586-gnu
I've been experiencing #34978 with these two targets. This applies the
hack in #35178 to these targets as well.
r? @alexcrichton
The arm-unknown-linux-musleabi target used in meta-rust for Yocto didn't
explicitly set the arch to ARMv6 and soft float but was instead done via
target spec files and never had the compiler running on the target.
Add -mrelax-relocations=no hacks to fix musl build
* this is just a start, dunno if it will work, but I'll just push it out to get feedback (my rust is still building 😢)
* I don't know much about rustbuild, so i just added that flag in there. it's a total hack, don't judge me
* I suspect the places in the musl .mk files are sufficient (but we may also need it present when building std), I'm not sure, needs more testing.
LLVM upgrade
As discussed in https://internals.rust-lang.org/t/need-help-with-emscripten-port/3154/46 I'm trying to update the used LLVM checkout in Rust.
I basically took @shepmaster's code and applied it on top (though I did the commits manually, the [original commits have better descriptions](https://github.com/rust-lang/rust/compare/master...avr-rust:avr-support).
With these changes I was able to build rustc. `make check` throws one last error on `run-pass/issue-28950.rs`. Output: https://gist.github.com/badboy/bcdd3bbde260860b6159aa49070a9052
I took the metadata changes as is and they seem to work, though it now uses the module in another step. I'm not sure if this is the best and correct way.
Things to do:
* [x] ~~Make `run-pass/issue-28950.rs` pass~~ unrelated
* [x] Find out how the `PositionIndependentExecutable` setting is now used
* [x] Is the `llvm::legacy` still the right way to do these things?
cc @brson @alexcrichton
Add ARM MUSL targets
Rebase of #33189.
I tested this by producing a std for `arm-unknown-linux-musleabi` then I cross compiled Hello world to said target. Checked that the produced binary was statically linked and verified that the binary worked under QEMU.
This depends on rust-lang/libc#341. I'll have to update this PR after that libc PR is merged.
I'm also working on generating ARM musl cross toolchain via crosstool-ng. Once I verified those work, I'll send a PR to rust-buildbot.
r? @alexcrichton
cc @timonvo
The targets are:
- `arm-unknown-linux-musleabi`
- `arm-unknown-linux-musleabihf`
- `armv7-unknown-linux-musleabihf`
These mirror the existing `gnueabi` targets.
All of these targets produce fully static binaries, similar to the
x86 MUSL targets.
For now these targets can only be used with `--rustbuild` builds, as
https://github.com/rust-lang/compiler-rt/pull/22 only made the
necessary compiler-rt changes in the CMake configs, not the plain
GNU Make configs.
I've tested these targets GCC 5.3.0 compiled again musl-1.1.12
(downloaded from http://musl.codu.org/). An example `./configure`
invocation is:
```
./configure \
--enable-rustbuild
--target=arm-unknown-linux-musleabi \
--musl-root="$MUSL_ROOT"
```
where `MUSL_ROOT` points to the `arm-linux-musleabi` prefix.
Usually that path will be of the form
`/foobar/arm-linux-musleabi/arm-linux-musleabi`.
Usually the cross-compile toolchain will live under
`/foobar/arm-linux-musleabi/bin`. That path should either by added
to your `PATH` variable, or you should add a section to your
`config.toml` as follows:
```
[target.arm-unknown-linux-musleabi]
cc = "/foobar/arm-linux-musleabi/bin/arm-linux-musleabi-gcc"
cxx = "/foobar/arm-linux-musleabi/bin/arm-linux-musleabi-g++"
```
As a prerequisite you'll also have to put a cross-compiled static
`libunwind.a` library in `$MUSL_ROOT/lib`. This is similar to [how
the x86_64 MUSL targets are built]
(https://doc.rust-lang.org/book/advanced-linking.html).
Add MIR Optimization Tests
I've starting working on the infrastructure for testing MIR optimizations.
The plan now is to have a set of test cases (written in Rust), compile them with -Z dump-mir, and check the MIR before and after each pass.
The compiler-rt build system has been a never ending cause of pain for Rust
unfortunately:
* The build system is very difficult to invoke and configure to only build
compiler-rt, especially across platforms.
* The standard build system doesn't actually do what we want, not working for
some of our platforms and requiring a significant number of patches on our end
which are difficult to apply when updating compiler-rt.
* Compiling compiler-rt requires LLVM to be compiled, which... is a big
dependency! This also means that over time compiler-rt is not guaranteed to
build against older versions of LLVM (or newer versions), and we often want to
work with multiple versions of LLVM simultaneously.
The makefiles and rustbuild already know how to compile C code, the code here is
far from the *only* C code we're compiling. This patch jettisons all logic to
work with compiler-rt's build system and just goes straight to the source. We
just list all files manually (copied from compiler-rt's
lib/builtins/CMakeLists.txt) and compile them into an archive.
It's likely that this means we'll fail to pick up new files when we upgrade
compiler-rt, but that seems like a much less significant cost to pay than what
we're currently paying.
cc #34400, first steps towards that
llvm, rt: build using the Ninja generator if available
The Ninja generator generally builds much faster than make. It may also
be used on Windows to have a vast speed improvement over the Visual
Studio generators.
Currently hidden behind an `--enable-ninja` flag because it does not
obey the top-level `-j` or `-l` flags given to `make`.
If local-rust is the same as the current version, then force a local-rebuild
In Debian, we would like the option to build/rebuild the current release from
*either* the current or previous stable release. So we use enable-local-rust
instead of enable-local-rebuild, and read the bootstrap key dynamically from
whatever is installed locally.
In general, it does not make much sense to allow enable-local-rust without also
setting the bootstrap key, since the build would fail otherwise.
(The way I detect "the bootstrap key of [the local] rustc installation" is a bit hacky, suggestions welcome.)
Soon the LLVM upgrade (#34743) will require an updated CMake installation, and
the easiest way to do this was to upgrade the Ubuntu version of the bots to
16.04. This in turn brings in a new MIPS compiler on the linux-cross builder,
which is now from the "official" ubuntu repositories. Unfortunately these
new compilers don't support compiling with the `-msoft-float` flag like we're
currently passing, causing compiles to fail.
This commit removes these flags as it's not clear *why* they're being passed, as
the mipsel targets also don't have it. At least if it's not supported by a
debian default compiler, perhaps it's not too relevant to support?
