ci: add a pr builder to test tools when submodules are updated
This PR adds the x86_64-gnu-tools builders to PRs where submodules are updated.
Since it's not possible to *start* the builder only when submodule changes are detected, I opted into adding a "decider" task at the start of the job which sets the `SKIP_JOB` environment variable when submodules are not updated, and I gated the most time-consuming tasks (the actual build and artifacts upload) on the variable not being there. All of this is conditionally included in the `steps/run.yml` only when a template parameter is present, so it should only affect that builder on PRs.
The cost for this should be a dummy builder running for 2/3 minutes for each PR, and we should be able to handle it.
Fixes https://github.com/rust-lang/rust/issues/61837
r? @alexcrichton
During installation of mingw, at least, the git directories change, so
we need to reset the core.autocrlf config to false.
Once we finish checking out submodules, check that the line endings are
\n and not \r\n.
The Azure image enables CRLF conversion on Windows builders, but that
caused regressions both in our test suite (the miri test suite broke)
and in the ecosystem, since we started shipping install scripts with
CRLF endings instead of the old LF. The Godbolt Compiler Explorer is one
such case of breakage.
This adds a step to the build explicitly disabling the conversion before
the repository is checked out.
Currently just run it through its paces but don't actually push to
official locations. Instead let's just push to a separate fork (mine) as
well as open issues in a separate fork (mine). Make sure that people
aren't pinged for these issues as well!
This should hopefully ensure that everything is working on Azure and
give us a chance to work through any issues that come up.
ci: Collect CPU usage statistics on Azure
This commit adds a script which we'll execute on Azure Pipelines which
is intended to run in the background and passively collect CPU usage
statistics for our builders. The intention here is that we can use this
information over time to diagnose issues with builders, see where we can
optimize our build, fix parallelism issues, etc. This might not end up
being too useful in the long run but it's data we've wanted to collect
for quite some time now, so here's a stab at it!
Comments about how this is intended to work can be found in the python
script used here to collect CPU usage statistics.
Closes#48828
This commit adds a script which we'll execute on Azure Pipelines which
is intended to run in the background and passively collect CPU usage
statistics for our builders. The intention here is that we can use this
information over time to diagnose issues with builders, see where we can
optimize our build, fix parallelism issues, etc. This might not end up
being too useful in the long run but it's data we've wanted to collect
for quite some time now, so here's a stab at it!
Comments about how this is intended to work can be found in the python
script used here to collect CPU usage statistics.
Closes#48828
Looks like Azure has updated images recently to install Rust by default,
but that can interfere with our own compiler (for example Cargo's test
suite we think) so be sure to uninstall it before proceeding.
Speed up Azure CI installing Windows dependencies
There is known issue where PowerShell is unreasonably slow downloading
files due to an issue with rendering the progress bar, see this [issue](https://github.com/PowerShell/PowerShell/issues/2138)
That issue is fixed in PowerShell Core (available in Azure Pipelines as
pwsh.exe) but it can also be worked around by setting:
$ProgressPreference = 'SilentlyContinue'
I measured downloading LLVM and it took about 220s before, 5s after, so the improvement is significant.
ci: Favor SCRIPT instead of RUST_CHECK_TARGET
Since #61212 we've been timing out on OSX, and this looks to be because
we're building tools like Cargo and the RLS twice instead of once. This
turns out to be a slight bug in our configuration. CI builders using the
`RUST_CHECK_TARGET` directive actually execute `make all` just before
their acual target. In `make all` we're building a stage2 cargo, and
then in `make dist` we're building a stage1 cargo.
Other builders use `SCRIPT` which provides explicit control over what
`x.py` script, for example, is used to execute the build. This moves
almost all targets to using `SCRIPT` to ensure that we're explicitly
specifying what's being built where. Additionally this updates the logic
of `RUST_CHECK_TARGET` to remove the pre-flight tidy as well as the
pre-flight `make all`. The system LLVM builder (run on PRs) now
explicitly runs tidy first and then runs the rest of the test suite.
Since #61212 we've been timing out on OSX, and this looks to be because
we're building tools like Cargo and the RLS twice instead of once. This
turns out to be a slight bug in our configuration. CI builders using the
`RUST_CHECK_TARGET` directive actually execute `make all` just before
their acual target. In `make all` we're building a stage2 cargo, and
then in `make dist` we're building a stage1 cargo.
Other builders use `SCRIPT` which provides explicit control over what
`x.py` script, for example, is used to execute the build. This moves
almost all targets to using `SCRIPT` to ensure that we're explicitly
specifying what's being built where. Additionally this updates the logic
of `RUST_CHECK_TARGET` to remove the pre-flight tidy as well as the
pre-flight `make all`. The system LLVM builder (run on PRs) now
explicitly runs tidy first and then runs the rest of the test suite.
There is known issue where PowerShell is unreasonably slow downloading
files due to an issue with rendering the progress bar, see:
https://github.com/PowerShell/PowerShell/issues/2138
That issue is fixed in PowerShell Core (available in Azure Pipelines as
pwsh.exe) but it can also be worked around by setting:
$ProgressPreference = 'SilentlyContinue'
* Clean up the `install-windows-build-deps.yml` file and add some more
comments where appropriate.
* Add some comments to `run.yml`
* Don't fast path the `rustfmt` submodule, but we'll take care of that
later if necessary.
This commit is intended to go through and review/refactor the azure
pipelines configuration we have. The major changes are:
* The separate `{windows,macos,linux}.yml` files are now all merged into
one `run.yml`. This allows a shared "master flow" for all platforms
with divergence only where necessary.
* Some install steps have been separated as `install-*.yml` scripts,
where each script internally matches on the appropriate OS and then
delegates accordingly.
* Some various bits and pieces of cruft have been removed which were
artifacts of Travis's setup or similar.
