Commit Graph

200 Commits

Author SHA1 Message Date
Alex Crichton
2727866831
travis: Don't enable quiet tests
This makes travis problems more difficult to debug, so let's just enable more
verbose logging.
2017-03-21 16:51:57 -04:00
Alex Crichton
b5c35c5f7a
travis: Add timestamps to all build messages
When debugging why builds are taking so long it's often useful to get the
timestamp of all log messages as we're not always timing every tiny step of the
build. I wrote a [utility] for prepending a relative timestamp from the start of
a process which is now downloaded to the builders and is what we wrap the entire
build invocation in.

[utility]: https://github.com/alexcrichton/stamp-rs

Closes #40577
2017-03-21 16:51:57 -04:00
Alex Crichton
7305ca3852 travis: Don't set RUST_LOG globally
I have a suspicion that this caused a large regression in cycle times by forcing
the compiler to perform more checks on every `debug!` statement, so let's test
this out by removing the `RUST_LOG` env var globally.

This regression in cycle time was witnessed between [two] [builds] where the
[PR] in question didn't do much suspicious. Judging by how the stage0 times
*also* regressed though then this is my best guess.

[two]: https://travis-ci.org/rust-lang/rust/builds/210149932
[builds]: https://travis-ci.org/rust-lang/rust/builds/210179995
[PR]: https://github.com/rust-lang/rust/pull/40446
2017-03-20 14:32:36 -07:00
bors
703b246287 Auto merge of #40482 - wagenet:fix-old-macos, r=alexcrichton
Target builds for older macOS

Fix #40481
2017-03-18 00:40:10 +00:00
Alex Crichton
ec829fe00e travis: Update sccache binary
I've built a local copy with mozilla/sccache#79 and mozilla/sccache#78. Let's
see if that helps #40240!
2017-03-16 07:55:15 -07:00
Peter Wagenet
61d93877b7 Target builds for older macOS 2017-03-14 10:08:29 -07:00
Alex Crichton
b5798a9be8 Update Cargo to fix nightly channel
This commit updates Cargo with rust-lang/cargo#3820 which includes a fix for
rust-lang/cargo#3819. At the same time this also slightly tweaks how rustbuild
builds cargo to ensure that all the build information (including git info and
such) makes its way into the binary.

Closes rust-lang/cargo#3820
2017-03-12 10:18:16 -07:00
Alex Crichton
c60a58b6d1 Attempt to debug sccache in more locations
This should hopefully add support for debugging OSX and Windows presumed sccache
failures instead of just the Linux ones.
2017-03-11 09:07:50 -08:00
bors
1727b23317 Auto merge of #40432 - alexcrichton:rollup, r=alexcrichton
Rollup of 38 pull requests

- Successful merges: #39202, #39820, #39918, #39921, #40092, #40146, #40199, #40225, #40239, #40257, #40259, #40261, #40277, #40278, #40287, #40297, #40311, #40315, #40319, #40324, #40336, #40340, #40344, #40345, #40367, #40369, #40372, #40373, #40379, #40385, #40386, #40389, #40400, #40404, #40410, #40422, #40423, #40424
- Failed merges: #40220, #40329, #40426
2017-03-11 05:41:41 +00:00
Alex Crichton
d03b6b3ea0 Rollup merge of #40424 - alexcrichton:faster-travis-osx, r=brson
travis: Remove compiling OpenSSL through homebrew

I don't believe that we need this any more now that `cargo-vendor` isn't
installed to create a source tarball (that only happens on Linux)
2017-03-10 20:05:06 -08:00
Alex Crichton
c1c3b09f28 Rollup merge of #40324 - alexcrichton:sccache-errors, r=aturon
travis: Attempt to debug sccache failures

I can't find anything that'd cause unexpected EOF in the source, so let's try
taking a look at the error logs on failures.
2017-03-10 16:18:28 -08:00
Alex Crichton
dcfc7ee853 travis: Remove compiling OpenSSL through homebrew
I don't believe that we need this any more now that `cargo-vendor` isn't
installed to create a source tarball (that only happens on Linux)
2017-03-10 10:25:16 -08:00
Alex Crichton
993eae1816 rustc: Support auto-retry linking on a segfault
This is a last-ditch attempt to help our pain with dealing with #38878 on the
bots. A new environment variable is added to the compiler,
`RUSTC_RETRY_LINKER_ON_SEGFAULT`, which will instruct the compiler to
automatically retry the final linker invocation if it looks like the linker
segfaulted (up to 2 extra times).

Unfortunately there have been no successful attempts to debug #38878. The only
information seems to be that the linker (e.g. `ld` on OSX) is segfaulting
somewhere in some thread pool implementation. This appears to be spurious as
failed PRs will later merge.

The hope is that this helps the queue keep moving without clogging and delaying
PRs due to #38878.
2017-03-10 09:13:53 -08:00
Petr Hosek
9a8461104e travis: Fuchsia builder
This change introduces a Dockerfile and script which builds a complete
Fuchsia toolchain which can be used to build Rust distribution for
Fuchsia. We only support cross-compiling at the moment, hence only
setting the target.
2017-03-10 08:28:40 -08:00
Alex Crichton
a8cacd3d21 travis: Attempt to debug sccache failures
I can't find anything that'd cause unexpected EOF in the source, so let's try
taking a look at the error logs on failures.
2017-03-10 07:51:14 -08:00
Alex Crichton
f44801c5ee travis: Split the linux-tested-targets builder
Travis only gives us 30GB disk space and we don't currently have an option to
increase that. Each musl target generates "hello world" binaries of about 3.5MB
in size, and we're testing two targets in the same image. We have around 3k
run-pass tests and 2 musl targets which works out to around 20GB. That's
dangerously close to the limit and is causing PRs to bounce.

This PR splits up the builder in two, one for x86_64 musl and the other for
i686. Hopefully that'll keep us under the disk limit.

Closes #40359
2017-03-09 11:54:24 -08:00
Ariel Ben-Yehuda
8bbbfec96f add some disk usage accounting
Try to figure out why we are out of free space
2017-03-09 00:05:36 +02:00
Alex Crichton
500076e141 travis: Split Android into dist/test images
PRs can't land againt beta right now because the android bot is filling up on
disk space. I don't really know what's going on but the android bot is the
longest one to run anyway so it'll benefit from being split up regardless.
2017-02-27 21:20:23 -08:00
Alex Crichton
65b46098b7 appveyor: Use sccache on pc-windows-gnu for caching
Now that mozilla/sccache#43 is fixed the caching works for MinGW on Windows. We
still can't use it for MSVC just yet, but I'll try to revive that branch at some
point.
2017-02-27 11:51:44 -08:00
Alex Crichton
c08f3824cd travis: Make more network requests retryable
This commit attempts to move more network operations to being retryable through
various operations. For example git submodule updates, downloading snapshots,
etc, are now all in retryable steps.

Hopefully this commit can cut down on the number of network failures we've been
seeing!
2017-02-25 21:28:54 -08:00
Alex Crichton
255a87499d Update sccache binaries on CI
Currently CI builds can fail spuriously during the LLVM build (#39003). I
believe this is due to sccache, and I believe that in turn was due to the fact
that the sccache server used to just be a raw mio server. Historically raw mio
servers are quite complicated to get right, but this is why we built Tokio! The
sccache server has been migrated to Tokio which I suspect would fix any latent
issues.

I have no confirmation of this (never been able to reproduce the deadlock
locally), but my hunch is that updating sccache to the master branch will fix
the timeouts during the LLVM build.

The binaries previously came from Gecko's infrastructure, but I've built new
ones by hand for Win/Mac/Linux and uploaded them to our CI bucket.
2017-02-24 13:16:54 -08:00
Corey Farwell
c9737af4ed Rollup merge of #39754 - alexcrichton:less-assertions, r=brson
travis: Add builders without assertions

This commit adds three new builders, one OSX, one Linux, and one MSVC, which
will produce "nightlies" with LLVM assertions disabled. Currently all nightly
releases have LLVM assertions enabled to catch bugs before they reach the
beta/stable channels. The beta/stable channels, however, do not have LLVM
assertions enabled.

Unfortunately though projects like Servo are stuck on nightlies for the near
future at least and are also suffering very long compile times. The purpose of
this commit is to provide artifacts to these projects which are not distributed
through normal channels (e.g. rustup) but are provided for developers to use
locally if need be.

Logistically these builds will all be uploaded to `rustc-builds-alt` instead of
the `rustc-builds` folder of the `rust-lang-ci` bucket. These builds will stay
there forever (until cleaned out if necessary) and there are no plans to
integrate this with rustup and/or the official release process.
2017-02-14 10:07:31 -05:00
Eduard-Mihai Burtescu
d29f0bc8fa Automatically vendor Cargo deps when building the source tarballs. 2017-02-14 01:52:03 +02:00
Alex Crichton
0340ddeb3b travis: Add builders without assertions
This commit adds three new builders, one OSX, one Linux, and one MSVC, which
will produce "nightlies" with LLVM assertions disabled. Currently all nightly
releases have LLVM assertions enabled to catch bugs before they reach the
beta/stable channels. The beta/stable channels, however, do not have LLVM
assertions enabled.

Unfortunately though projects like Servo are stuck on nightlies for the near
future at least and are also suffering very long compile times. The purpose of
this commit is to provide artifacts to these projects which are not distributed
through normal channels (e.g. rustup) but are provided for developers to use
locally if need be.

Logistically these builds will all be uploaded to `rustc-builds-alt` instead of
the `rustc-builds` folder of the `rust-lang-ci` bucket. These builds will stay
there forever (until cleaned out if necessary) and there are no plans to
integrate this with rustup and/or the official release process.
2017-02-11 17:38:09 -08:00
Corey Farwell
6fb57bf13f Rollup merge of #39431 - alexcrichton:no-more-makefiles, r=brson
Delete the makefile build system

This PR deletes the makefile build system in favor of the rustbuild build system. The beta has now been branched so 1.16 will continue to be buildable from the makefiles, but going forward 1.17 will only be buildable with rustbuild.

Rustbuild has been the default build system [since 1.15.0](https://github.com/rust-lang/rust/pull/37817) and the makefiles were [proposed for deletion](https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368) at this time back in November of last year.

And now with the deletion of these makefiles we can start getting those sweet sweet improvements of using crates.io crates in the compiler!
2017-02-07 22:54:25 -05:00
Corey Farwell
370b63f386 Rollup merge of #39400 - alexcrichton:arm-cross-test, r=brson
Add support for test suites emulated in QEMU

This commit adds support to the build system to execute test suites that cannot
run natively but can instead run inside of a QEMU emulator. A proof-of-concept
builder was added for the `arm-unknown-linux-gnueabihf` target to show off how
this might work.

In general the architecture is to have a server running inside of the emulator
which a local client connects to. The protocol between the server/client
supports compiling tests on the host and running them on the target inside the
emulator.

Closes #33114
2017-02-07 22:54:23 -05:00
Alex Crichton
ce4abc3515 Delete Travis/AppVeyor makefile builders
We no longer need these builders as we're no longer testing the old build
system.
2017-02-06 08:42:54 -08:00
bors
f45992b300 Auto merge of #39415 - alexcrichton:fix-upload-dirs, r=brson
travis: Really delete the `doc` folder

Got two location to look at, be sure to delete them both.
2017-02-03 09:13:06 +00:00
bors
0c85f2a1bd Auto merge of #38847 - michaelwoerister:gate-on-incr-comp, r=alexcrichton
travis: Gate on some minimal support for incremental compilation.

This commit adds a travis job that

1. builds a stage2 compiler in incremental mode (but with empty incremental compilation cache), and
2. builds and runs the run-pass test suite also in incremental mode.

Building incrementally with an empty cache makes sure that the compiler doesn't crash in dependency tracking during bootstrapping. Executing the incrementally built test suite gives some measure of confidence that we generate valid code.

Note, however, that the above does not give strong guarantees about the validity of incremental compilation, it just provides a basis for being able to rely on from-scratch incr. comp. builds as reference values in further tests (which then do actual incremental compilation).

r? @alexcrichton
2017-01-31 06:13:05 +00:00
Alex Crichton
775b32305f travis: Really delete the doc folder
Got two location to look at, be sure to delete them both.
2017-01-30 15:21:35 -08:00
Alex Crichton
45d203df30 travis: Tweak artifact uploads
* Don't upload `*.wixpdb` files by accident
* Don't upload `doc` dir by accident
* Fix level of indirection on Travis
2017-01-30 08:56:30 -08:00
Alex Crichton
1747ce25ad Add support for test suites emulated in QEMU
This commit adds support to the build system to execute test suites that cannot
run natively but can instead run inside of a QEMU emulator. A proof-of-concept
builder was added for the `arm-unknown-linux-gnueabihf` target to show off how
this might work.

In general the architecture is to have a server running inside of the emulator
which a local client connects to. The protocol between the server/client
supports compiling tests on the host and running them on the target inside the
emulator.

Closes #33114
2017-01-29 14:16:41 -08:00
Alex Crichton
ac1e92328a Rollup merge of #39302 - alexcrichton:upload-all, r=brson
travis: Upload all artifacts in build/dist

Previously we only uploaded tarballs, but this modifies Travis/AppVeyor to
upload everything. We shouldn't have anything else in there to worry about and
otherwise we need to be sure to pick up pkg/msi/exe installers.
2017-01-27 16:41:50 -08:00
Alex Crichton
7095a48bf6 travis: Turn off core dumps on OSX
I've seen these take up quite a bit of log space and I have the sneaking
suspicion that they're just making our test suite take longer (sometimes timing
out on 32-bit OSX now). In any case the backtraces haven't proven too useful,
unfortunately.
2017-01-26 09:04:59 -08:00
Alex Crichton
8944582d1d travis: Upload all artifacts in build/dist
Previously we only uploaded tarballs, but this modifies Travis/AppVeyor to
upload everything. We shouldn't have anything else in there to worry about and
otherwise we need to be sure to pick up pkg/msi/exe installers.
2017-01-25 11:55:40 -08:00
Alex Crichton
f3dfcae202 rustbuild: Start building --enable-extended
This commit adds a new flag to the configure script,
`--enable-extended`, which is intended for specifying a desire to
compile the full suite of Rust tools such as Cargo, the RLS, etc. This
is also an indication that the build system should create combined
installers such as the pkg/exe/msi artifacts.

Currently the `--enable-extended` flag just indicates that combined
installers should be built, and Cargo is itself not compiled just yet
but rather only downloaded from its location. The intention here is to
quickly get to feature parity with the current release process and then
we can start improving it afterwards.

All new files in this PR inside `src/etc/installer` are copied from the
rust-packaging repository.
2017-01-24 14:48:03 -08:00
Alex Crichton
72c3148bb3 More test fixes from rollup 2017-01-20 13:49:16 -08:00
Alex Crichton
5e8d7a4b7c Merge branch 'older-glibc' into rollup 2017-01-20 08:36:50 -08:00
Alex Crichton
a759406dce Rollup merge of #39167 - alexcrichton:no-more-sha, r=brson
travis: Stop uploading sha256 files

We'll generate these later in the build process and otherwise they could just
cause spurious failures with files overwriting one another.

cc #38531
2017-01-20 08:35:49 -08:00
Alex Crichton
254380e2f3 Rollup merge of #39146 - alexcrichton:fix-osx-debug, r=michaelwoerister
travis: Fix post-failure lldb invocation

Pass an absolute path, not just the basename.
2017-01-20 08:35:48 -08:00
Alex Crichton
dd5d85ea76 Rollup merge of #39114 - alexcrichton:fix-osx-image, r=brson
travis: Tweak OSX image configuration

Somewhere between https://travis-ci.org/rust-lang/rust/jobs/192352185 and
https://travis-ci.org/rust-lang/rust/jobs/192440181 it looks like our
configuration for a newer OSX image was lost as LLDB has reverted itself back to
350. This fix appeared to work for the libc crate so let's see if we can
configure it to work for the rust repo as well.
2017-01-20 08:35:47 -08:00
Alex Crichton
4a298dde72 Rollup merge of #39111 - alexcrichton:more-cross-targets, r=brson
travis: Expand the `cross` linux image

This expands the `cross` travis matrix entry with a few more targets that our
nightlies are building:

* x86_64-rumprun-netbsd
* arm-unknown-linux-musleabi
* arm-unknown-linux-musleabihf
* armv7-unknown-linux-musleabihf
* mips-unknown-linux-musl
* mipsel-unknown-linux-musl

This commit doesn't compile custom toolchains like our current cross-image does,
but instead compiles musl manually and then compiles libunwind manually (like
x86_64) for use for the ARM targets and just uses openwrt toolchains for the
mips targets.

cc #38531
2017-01-20 08:35:46 -08:00
Alex Crichton
0a7420acd1 travis: Move glibc backwards in time
This commit updates the compilers for many of the artifacts that we're producing
on Travis. These compilers are all compiled by crosstool-ng as they're currently
done for the images in which we're building all our cross compiled compilers.

The purpose of this commit is that when we ship binaries the artifacts won't
require a newer glibc, but rather be as compatible as possible with Linux
distributions by working with a very old version of glibc.

This commit always allocates a new matrix entry for the i686/x86_64 builder.
This builder is dedicated to just producing artifacts and eventually we'll
expand it to building other tools like Cargo and the RLS. The other builders
testing i686 and x86_64 won't use these historical toolchains.
2017-01-19 14:32:26 -08:00
Alex Crichton
c457b819d7 travis: Stop uploading sha256 files
We'll generate these later in the build process and otherwise they could just
cause spurious failures with files overwriting one another.
2017-01-18 16:10:05 -08:00
Michael Woerister
5f118b954e travis: Gate on some minimal support for incremental compilation.
This commit adds a travis job that builds a stage2 compiler in
incremental mode (but with empty incremental compilation cache).
Building incrementally with an empty cache makes sure that the
compiler doesn't crash in dependency tracking during bootstrapping.
2017-01-18 12:06:22 -05:00
Alex Crichton
92e98a06d4 travis: Fix post-failure lldb invocation
Pass an absolute path, not just the basename.
2017-01-17 18:55:24 -08:00
Alex Crichton
6b23cc48db travis: Expand the cross linux image
This expands the `cross` travis matrix entry with a few more targets that our
nightlies are building:

* x86_64-rumprun-netbsd
* arm-unknown-linux-musleabi
* arm-unknown-linux-musleabihf
* armv7-unknown-linux-musleabihf
* mips-unknown-linux-musl
* mipsel-unknown-linux-musl

This commit doesn't compile custom toolchains like our current cross-image does,
but instead compiles musl manually and then compiles libunwind manually (like
x86_64) for use for the ARM targets and just uses openwrt toolchains for the
mips targets.
2017-01-16 18:50:01 -08:00
Alex Crichton
baff51c040 travis: Tweak OSX image configuration
Somewhere between https://travis-ci.org/rust-lang/rust/jobs/192352185 and
https://travis-ci.org/rust-lang/rust/jobs/192440181 it looks like our
configuration for a newer OSX image was lost as LLDB has reverted itself back to
350. This fix appeared to work for the libc crate so let's see if we can
configure it to work for the rust repo as well.
2017-01-16 18:43:38 -08:00
Alex Crichton
6162637762 travis: Add i586 linux and i686 musl
This commit expands the existing x86_64-musl entry in the Travis matrix to also
build/test i586-unknown-linux-gnu and i686-unknown-linux-musl.

cc #38531
Closes #39053
2017-01-15 18:40:57 -08:00
bors
ff591b6dc0 Auto merge of #39042 - alexcrichton:upload-more, r=brson
travis: Expand dist builder coverage

This commit adds six new travis matrix entires for doing cross-compiled
distribution builds of the compiler. The support added in #38731 allows us to
quickly compile a complete suite of distribution artifacts for cross-compiled
platforms, and currently each matrix entry (when fully cached) clocks in around
an hour to finish. Note that a full test run typically takes about two hours
right now.

With further optimizations coming down the pike in #39026 this commit also
starts doubling up cross-compiled distribution builders on each matrix entry. We
initially planned to do one build per entry, but it's looking like we may be
able to get by with more than one in each entry. Depending on how long these
builds take we may even be able to up it to three, but we'll start with two
first.

This commit then completes the suite of cross-compiled compilers that we're
going to compile, adding it for a whole litany of platforms detailed in the
changes to the docker files here. The existing `cross` image is also trimmed
down quite a bit to avoid duplicate work, and we'll eventually provision it for
far more cross compilation as well.

Note that the gcc toolchains installed to compile most of these compilers are
inappropriate for actualy distribution. The glibc they pull in is much newer
than we'd like, so before we turn nightlies off we'll need to tweak these docker
files to custom build toolchains like the current `linux-cross` docker image
does.
2017-01-15 23:49:24 +00:00
Alex Crichton
a6d88b023a travis: Expand dist builder coverage
This commit adds six new travis matrix entires for doing cross-compiled
distribution builds of the compiler. The support added in #38731 allows us to
quickly compile a complete suite of distribution artifacts for cross-compiled
platforms, and currently each matrix entry (when fully cached) clocks in around
an hour to finish. Note that a full test run typically takes about two hours
right now.

With further optimizations coming down the pike in #39026 this commit also
starts doubling up cross-compiled distribution builders on each matrix entry. We
initially planned to do one build per entry, but it's looking like we may be
able to get by with more than one in each entry. Depending on how long these
builds take we may even be able to up it to three, but we'll start with two
first.

This commit then completes the suite of cross-compiled compilers that we're
going to compile, adding it for a whole litany of platforms detailed in the
changes to the docker files here. The existing `cross` image is also trimmed
down quite a bit to avoid duplicate work, and we'll eventually provision it for
far more cross compilation as well.

Note that the gcc toolchains installed to compile most of these compilers are
inappropriate for actualy distribution. The glibc they pull in is much newer
than we'd like, so before we turn nightlies off we'll need to tweak these docker
files to custom build toolchains like the current `linux-cross` docker image
does.
2017-01-15 10:14:43 -08:00
bors
20ba64d2b4 Auto merge of #39055 - aidanhs:aphs-deinit-before-init, r=alexcrichton
If submodule init fails, try from scratch

See #39051

I wonder if the cause could be some strange not-quite-checked-out state in a submodule. Try and fix this by force deinitialising everything before initialising (this will not throw away downloaded objects, git will skip them on the next attempt at cloning).

r? @alexcrichton
2017-01-15 13:00:49 +00:00
bors
b13cc05c48 Auto merge of #39021 - alexcrichton:try-debug-travis, r=brson
travis: Attempt to debug OSX linker segfaults

This commit attempts to debug the segfaults that we've been seeing on OSX on
Travis. I have no idea what's going on here mostly, but let's try to look at
core dumps and get backtraces to see what's going on. This commit itself is
mostly a complete shot in the dark, I'm not sure if this even works...

cc #38878
2017-01-14 04:29:44 +00:00
Aidan Hobson Sayers
7b6c2cbe56 If submodule init fails, try from scratch 2017-01-14 03:10:45 +00:00
Alex Crichton
318767266f travis: Start uploading artifacts on commits
This commit starts adding the infrastructure for uploading release artifacts
from AppVeyor/Travis on each commit. The idea is that eventually we'll upload a
full release to AppVeyor/Travis in accordance with plans [outlined earlier].

Right now this configures Travis/Appveyor to upload all tarballs in the `dist`
directory, and various images are updated to actually produce tarballs in these
directories. These are nowhere near ready to be actual release artifacts, but
this should allow us to play around with it and test it out. Once this commit
lands we should start seeing artifacts uploaded on each commit.

[outlined earlier]: https://internals.rust-lang.org/t/rust-ci-release-infrastructure-changes/4489
2017-01-12 15:29:04 -08:00
Alex Crichton
6ae6160e42 travis: Attempt to debug OSX linker segfaults
This commit attempts to debug the segfaults that we've been seeing on OSX on
Travis. I have no idea what's going on here mostly, but let's try to look at
core dumps and get backtraces to see what's going on. This commit itself is
mostly a complete shot in the dark, I'm not sure if this even works...

cc #38878
2017-01-12 14:59:57 -08:00
Alex Crichton
882426b404 travis: Wrap submodules updates in travis_retry
Let's try to squash some of those network issues with a `travis_retry`
tool to just retry the command a few times.
2017-01-06 21:28:57 -08:00
Alex Crichton
1a040b36cb rustbuild: Quickly dist cross-host compilers
This commit optimizes the compile time for creating tarballs of cross-host
compilers and as a proof of concept adds two to the standard Travis matrix. Much
of this commit is further refactoring and refining of the `step.rs` definitions
along with the interpretation of `--target` and `--host` flags. This has gotten
confusing enough that I've also added a small test suite to
`src/bootstrap/step.rs` to ensure what we're doing works and doesn't regress.

After this commit when you execute:

    ./x.py dist --host $MY_HOST --target $MY_HOST

the build system will compile two compilers. The first is for the build platform
and the second is for the host platform. This second compiler is then packaged
up and placed into `build/dist` and is ready to go. With a fully cached LLVM and
docker image I was able to create a cross-host compiler in around 20 minutes
locally.

Eventually we plan to add a whole litany of cross-host entries to the Travis
matrix, but for now we're just adding a few before we eat up all the extra
capacity.

cc #38531
2017-01-04 11:41:16 -08:00
Alex Crichton
4781eb315b travis: Add a distcheck target
This commit adds a new entry to the Travis matrix which performs a "distcheck",
which basically means that we create a tarball, extract that tarball, and then
build/test inside there. This ensures that the tarballs we produce are actually
able to be built/tested!

Along the way this also updates the rustbuild distcheck definition to propagate
the configure args from the top-level invocation.

Closes #38691
2016-12-30 09:36:23 -08:00
Alex Crichton
3eb459ff5f Merge branch 'aux-tests' of https://github.com/alexcrichton/rust into rollup 2016-12-29 17:29:32 -08:00
Alex Crichton
031aa58d24 Rollup merge of #38655 - alexcrichton:travis-and-then, r=brson
travis: Use `&&` intead of `;`

Show errors sooner and try not to hide them behind lots of other walls of text.
2016-12-29 17:26:31 -08:00
Alex Crichton
9bb3543885 Rollup merge of #38631 - alexcrichton:supafast, r=brson
rustbuild: Compile rustc twice, not thrice

This commit switches the rustbuild build system to compiling the
compiler twice for a normal bootstrap rather than the historical three
times.

Rust is a bootstrapped language which means that a previous version of
the compiler is used to build the next version of the compiler. Over
time, however, we change many parts of compiler artifacts such as the
metadata format, symbol names, etc. These changes make artifacts from
one compiler incompatible from another compiler. Consequently if a
compiler wants to be able to use some artifacts then it itself must have
compiled the artifacts.

Historically the rustc build system has achieved this by compiling the
compiler three times:

* An older compiler (stage0) is downloaded to kick off the chain.
* This compiler now compiles a new compiler (stage1)
* The stage1 compiler then compiles another compiler (stage2)
* Finally, the stage2 compiler needs libraries to link against, so it
  compiles all the libraries again.

This entire process amounts in compiling the compiler three times.
Additionally, this process always guarantees that the Rust source tree
can compile itself because the stage2 compiler (created by a freshly
created compiler) would successfully compile itself again. This
property, ensuring Rust can compile itself, is quite important!

In general, though, this third compilation is not required for general
purpose development on the compiler. The third compiler (stage2) can
reuse the libraries that were created during the second compile. In
other words, the second compilation can produce both a compiler and the
libraries that compiler will use. These artifacts *must* be compatible
due to the way plugins work today anyway, and they were created by the
same source code so they *should* be compatible as well.

So given all that, this commit switches the default build process to
only compile the compiler two times, avoiding this third compilation
by copying artifacts from the previous one. Along the way a new entry in
the Travis matrix was also added to ensure that our full bootstrap can
succeed. This entry does not run tests, though, as it should not be
necessary.

To restore the old behavior of a full bootstrap (three compiles) you can
either pass:

    ./configure --enable-full-bootstrap

or if you're using config.toml:

    [build]
    full-bootstrap = true

Overall this will hopefully be an easy 33% win in build times of the
compiler. If we do 33% less work we should be 33% faster! This in turn
should affect cycle times and such on Travis and AppVeyor positively as
well as making it easier to work on the compiler itself.
2016-12-29 17:26:25 -08:00
Alex Crichton
a68f8866a7 Rollup merge of #38609 - alexcrichton:less-compress, r=japaric
travis: Don't use -9 on gzip

I timed this locally and plain old `gzip` took 2m06s while `gzip -9` took a
whopping 6m23s to save a mere 4MB out of 1.2GB. Let's shave a few minutes off
the Android builder by turning down the compression level.
2016-12-29 17:26:19 -08:00
Alex Crichton
900dd8a7b9 rustbuild: Don't run pretty tests by default
This commit relegates all pretty tests to not get run by default and rather get
run as part of an "aux" test suite. This "aux" suite is renamed from the old
"cargotest" suite to just collect tests that don't need to run everywhere but
should at least pass on Unix/Windows.
2016-12-29 09:55:16 -08:00
Alex Crichton
7046fea5be rustbuild: Compile rustc twice, not thrice
This commit switches the rustbuild build system to compiling the
compiler twice for a normal bootstrap rather than the historical three
times.

Rust is a bootstrapped language which means that a previous version of
the compiler is used to build the next version of the compiler. Over
time, however, we change many parts of compiler artifacts such as the
metadata format, symbol names, etc. These changes make artifacts from
one compiler incompatible from another compiler. Consequently if a
compiler wants to be able to use some artifacts then it itself must have
compiled the artifacts.

Historically the rustc build system has achieved this by compiling the
compiler three times:

* An older compiler (stage0) is downloaded to kick off the chain.
* This compiler now compiles a new compiler (stage1)
* The stage1 compiler then compiles another compiler (stage2)
* Finally, the stage2 compiler needs libraries to link against, so it
  compiles all the libraries again.

This entire process amounts in compiling the compiler three times.
Additionally, this process always guarantees that the Rust source tree
can compile itself because the stage2 compiler (created by a freshly
created compiler) would successfully compile itself again. This
property, ensuring Rust can compile itself, is quite important!

In general, though, this third compilation is not required for general
purpose development on the compiler. The third compiler (stage2) can
reuse the libraries that were created during the second compile. In
other words, the second compilation can produce both a compiler and the
libraries that compiler will use. These artifacts *must* be compatible
due to the way plugins work today anyway, and they were created by the
same source code so they *should* be compatible as well.

So given all that, this commit switches the default build process to
only compile the compiler three times, avoiding this third compilation
by copying artifacts from the previous one. Along the way a new entry in
the Travis matrix was also added to ensure that our full bootstrap can
succeed. This entry does not run tests, though, as it should not be
necessary.

To restore the old behavior of a full bootstrap (three compiles) you can
either pass:

    ./configure --enable-full-bootstrap

or if you're using config.toml:

    [build]
    full-bootstrap = true

Overall this will hopefully be an easy 33% win in build times of the
compiler. If we do 33% less work we should be 33% faster! This in turn
should affect cycle times and such on Travis and AppVeyor positively as
well as making it easier to work on the compiler itself.
2016-12-28 14:49:00 -08:00
Alex Crichton
88429dc575 travis: Use && intead of ;
Show errors sooner and try not to hide them behind lots of other walls of text.
2016-12-28 09:29:14 -08:00
Alex Crichton
dad0076569 travis: Update the OSX image we run tests in
The current image is `xcode7.3`, Travis's current default. Unfortunately this
has a version of LLDB which doesn't support debuginfo-lldb tests (see #32520),
so we're not running LLDB tests on Travis yet.

This switches us to the newest image from Travis, `xcode8.2`, which should have
a newer version of LLDB we can run tests against.
2016-12-26 15:40:22 -08:00
Alex Crichton
67cc77ccde travis: Don't use -9 on gzip
I timed this locally and plain old `gzip` took 2m06s while `gzip -9` took a
whopping 6m23s to save a mere 4MB out of 1.2GB. Let's shave a few minutes off
the Android builder by turning down the compression level.
2016-12-25 22:13:50 -08:00
bors
dea7ef3424 Auto merge of #38419 - alexcrichton:travis-osx-32-bit, r=brson
travis: Fix testing 32-bit OSX target

We passed --target when we meant to pass --build, meaning we tested only the
standard library for 32-bit, not the whole compiler like we intended.
2016-12-17 11:10:11 +00:00
Alex Crichton
0f742e600b travis: Fix testing 32-bit OSX target
We passed --target when we meant to pass --build, meaning we tested only the
standard library for 32-bit, not the whole compiler like we intended.
2016-12-16 16:32:40 -08:00
Alex Crichton
96a5fc76dc rustbuild: Add sccache support
This commit adds support for sccache, a ccache-like compiler which works on MSVC
and stores results into an S3 bucket. This also switches over all Travis and
AppVeyor automation to using sccache to ensure a shared and unified cache over
time which can be shared across builders.

The support for sccache manifests as a new `--enable-sccache` option which
instructs us to configure LLVM differently to use a 'sccache' binary instead of
a 'ccache' binary. All docker images for Travis builds are updated to download
Mozilla's tooltool builds of sccache onto various containers and systems.
Additionally a new `rust-lang-ci-sccache` bucket is configured to hold all of
our ccache goodies.
2016-12-14 15:40:18 -08:00
Alex Crichton
0e272de69f mk: Switch rustbuild to the default build system
This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2016-01-02.

[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368

This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.

Closes #37858
2016-12-07 00:30:23 -08:00
Eduard Burtescu
0268cfaa1f Set RUST_BACKTRACE=1 on travis to be more helpful. 2016-11-12 14:30:28 +02:00
Alex Crichton
008cc2d999 Move all Linux/OSX CI infastructure to Travis
This commit configures our `.travis.yml` to test the full suite of tests we have
on Buildbot right now. A whole mess of docker images are added to the `src/ci`
directory which represent all the build environments for each configuration.
Each of these environments is then configured in `.travis.yml` to run on the
auto branch.

Note that the full matrix of tests aren't intended to be run on all PRs.
Instead, we continue to run only one entry in the matrix, forcing all others to
finish quickly. Only the `auto` branch should run the full matrix of builds.

Also note that the infrastructure hasn't quite been allocated yet to the
rust-lang/rust repository, so everything is disabled for now except for the one
build that happens on PRs. Once that infrastructure is allocated though we can
enable this and let it fly!

Notable modifications from the current test suite today:

* Android tests are run in rustbuild instead of the makefiles, for whatever
  reason I couldn't get the makefiles to work on Travis.
* A debuginfo test was updated to work with the current version of the Android
  NDK.
* Some dependencies in `mk/tests.mk` were fixed to allow running tests in
  parallel.
2016-11-11 07:36:40 -08:00
Alex Crichton
31a8638e5e rustbuild: Tweak for vendored dependencies
A few changes are included here:

* The `winapi` and `url` dependencies were dropped. The source code for these
  projects is pretty weighty, and we're about to vendor them, so let's not
  commit to that intake just yet. If necessary we can vendor them later but for
  now it shouldn't be necessary.

* The `--frozen` flag is now always passed to Cargo, obviating the need for
  tidy's `cargo_lock` check.

* Tidy was updated to not check the vendor directory

Closes #34687
2016-11-08 07:32:05 -08:00
Corey Farwell
c8c6d2c732 Use quieter test output when running tests on Travis CI.
Fixes https://github.com/rust-lang/rust/issues/36788.
2016-10-30 17:31:17 -04:00
Corey Farwell
3a96fe3275 Transition Travis CI to use rustbuild. 2016-09-02 12:48:55 -04:00
Stefan Schindler
ca259666a5 Reduce git clone --depth from 50 to 1 2016-07-28 11:48:43 +02:00
Seo Sanghyeon
b1651fb4d2 Use Docker for Travis 2016-06-03 11:44:30 +09:00
arcnmx
1181ca4b51 Use Travis trusty infrastructure 2015-10-19 18:31:02 -04:00
Alex Crichton
27dd6dd3db Tweak Travis to use GCE
Travis CI has new infrastructure using the Google Compute Engine which has both
faster CPUs and more memory, and we've been encouraged to switch as it should
help our build times! The only downside currently, however, is that IPv6 is
disabled, causing a number of standard library tests to fail.

Consequently this commit tweaks our travis config in a few ways:

* ccache is disabled as it's not working on GCE just yet
* Docker is used to run tests inside which reportedly will get IPv6 working
* A system LLVM installation is used instead of building LLVM itself. This is
  primarily done to reduce build times, but we want automation for this sort of
  behavior anyway and we can extend this in the future with building from source
  as well if needed.
* gcc-specific logic is removed as the docker image for Ubuntu gives us a
  recent-enough gcc by default.
2015-09-29 16:56:35 -07:00
Huon Wilson
91a9260a16 Run tidy by itself on travis.
It is very difficult to find tidy problems in the midst of the output of
the LLVM/jemalloc/etc. build, and travis is great for the former, so
lets remove that problem.
2015-08-28 22:59:00 -07:00
Alexis Beingessner
b0acde7370 make travis extreme 2015-07-24 09:45:34 -07:00
Alexis Beingessner
e8a03285b7 Ratchet up travis to build stage1 and our own LLVM.
Tidy is still run first for failing fast on the easy stuff.

To accomplish this we have travis actually persist ccache across builds. This
has LLVM built within 6 minutes, and all of stage1 built within 18.
Caching should work on fresh PRs (cache acquired from the master branch).
Because all we persist is ccache, there is minimal danger of persisting corrupt
build state.

I had to mangle `configure` a bit to make --enable-ccache work when custom
compilers are provide via CC and CXX.
2015-07-15 10:48:50 -07:00
Matthew Astley
512b105b15 point to buildbot
Since 7b6ecc00
2015-06-11 19:57:16 +01:00
dan@daramos.com
637f2c5ea4 Allow travis to use newer-faster infrastructure for building. http://blog.travis-ci.com/2014-12-17-faster-builds-with-container-based-infrastructure/ 2015-01-01 02:00:29 -05:00
Alex Crichton
7b6ecc009c travis: Fix for real this time
I ended up botching the merge when making the rollup, and the fix was to just
not configure LLVM all via --llvm-root with a nonexistent path.
2014-10-03 07:25:10 -07:00
Alex Crichton
d911936dbd Merge branch 'travis' into rollup
Conflicts:
	.travis.yml
2014-10-02 21:02:36 -07:00
Alex Crichton
9ac804ecab travis: Stop building and testing rust
Instead, only run `make tidy`. The tidy script can run quite quickly, and it's
super annoying to run tests for 50 minutes only to have bors fail with a
"trailing whitespace" error.
2014-10-02 15:11:52 -07:00
Alex Crichton
cc9347a902 travis: Stop building and testing rust
Instead, only run `make tidy`. The tidy script can run quite quickly, and it's
super annoying to run tests for 50 minutes only to have bors fail with a
"trailing whitespace" error.
2014-10-02 13:50:43 -07:00
Alex Crichton
9af1b0e5e1 travis: Move from travis_wait to time-passes
It's looking like we're still timing out all over the place with travis_wait
because the entire `make -j4 rustc-stage1` command is taking too long. Instead,
achieve roughly the same idea by just having `-Z time-passes` printing
information. We shouldn't have a pass that takes longer than 10 minutes in
isolation.
2014-07-01 20:21:16 -07:00
Alex Crichton
8c057984c6 travis: Don't use a local jemalloc
Turns out they don't have the `je_` prefix, so we can't use the system-installed
jemalloc.
2014-06-12 16:12:37 -07:00
Alex Crichton
46014c7b14 travis: Use a pre-installed jemalloc 2014-06-12 00:28:01 -07:00
Alex Crichton
bc17897116 travis: Prevent timeouts with travis_wait
The most frequent failure for our travis builds is running into the timeout
limits when building the compiler itself. Building librustc takes a very long
amount of time, often hitting the 10 minutes with no output threshold that
travis imposes on us.

This commit switches the relevant `make` step to being wrapped in the
`travis_wait` command [1]. This command will print something once a minute so as
to not time out a build.

This will hopefully enable us to have fewer flaky builds on travis!

[1]: http://docs.travis-ci.com/user/build-timeouts/
2014-06-04 17:16:37 -07:00
Flavio Percoco
3c4edff9b5 Let travis check docs for stage1 2014-03-20 10:20:08 +01:00
Alex Crichton
37b3a60b78 travis: Use LLVM 3.4 instead of LLVM 3.3
This version is slightly more up to date and is closer to the 3.5 that we're
using. This also updates the travis config to have a build matrix which tests
rust against LLVM 3.3 and 3.4. For pull requests only LLVM 3.4 is tested to
reduce the load on travis.

This is mostly just fluff, there's no real reason to gate rust on these results,
it's more of just a nice thing to know when we break compatibility with LLVM 3.3
and 3.4 (and eventually 3.5). This turns off notifications of failed commits
(which are sent out for pushes to master).
2014-03-06 00:25:19 -08:00
Luca Bruno
357cadf722 travis: remove manually added RUSTFLAGS
Indirect dependencies should now be picked up through the
autogenerated llvmdeps.rs instead.

Signed-off-by: Luca Bruno <lucab@debian.org>
2014-03-02 20:02:42 +01:00
Alex Crichton
bbdaf01a5a travis: Use LLVM 3.3 from the official repos
We can be certain that this version of LLVM will not be changing, so we don't
have to worry about API drift over time.
2014-02-26 15:22:50 -08:00
Alex Crichton
a7b1d65080 Run the travis build as one large command
It appears that travis doesn't stop running script commands after the first one
fails (see https://github.com/travis-ci/travis-ci/issues/1066), so chain all our
commands together with && for now.
2014-02-24 21:22:27 -08:00
Alex Crichton
5bb204ffdb Add a Travis-CI configuration for the repo
Travis CI provides an easy-to-use continuous integration infrastructure for
github repos to use. Travis will automatically test all PRs which are opened
against the rust repository, informing PR owners of the test results.

I believe that this will be a very convenient piece of infrastructure as we'll
be able to reduce the load on bors quite a bit. In theory all PRs opened have
had the full test suite run against them, but unfortunately this is rarely the
case (I'm a prime suspect). Travis will be able to provide easy and relatively
quick (~30min) feedback for PRs. By ensuring fewer failures on bors, we can
hopefully feed more successful jobs to bors.

Overall, I expect this to be very helpful for new contributors as well as
regular contributors as it's another layer of tests being run which will
hopefully catch things sooner. One of the most convenient parts about using
Travis is that there's very little burden in terms of maintenance, and if things
go wrong we can easily turn travis completely off.

Note that this is *not* the metric by which a PR will be merged with. Using
travis will purely be another source for running tests, we will continue to gate
all PRs on bors.
2014-02-20 21:43:25 -08:00