ci: Collect CPU usage statistics on Azure
This commit adds a script which we'll execute on Azure Pipelines which
is intended to run in the background and passively collect CPU usage
statistics for our builders. The intention here is that we can use this
information over time to diagnose issues with builders, see where we can
optimize our build, fix parallelism issues, etc. This might not end up
being too useful in the long run but it's data we've wanted to collect
for quite some time now, so here's a stab at it!
Comments about how this is intended to work can be found in the python
script used here to collect CPU usage statistics.
Closes#48828
This commit adds a script which we'll execute on Azure Pipelines which
is intended to run in the background and passively collect CPU usage
statistics for our builders. The intention here is that we can use this
information over time to diagnose issues with builders, see where we can
optimize our build, fix parallelism issues, etc. This might not end up
being too useful in the long run but it's data we've wanted to collect
for quite some time now, so here's a stab at it!
Comments about how this is intended to work can be found in the python
script used here to collect CPU usage statistics.
Closes#48828
Looks like Azure has updated images recently to install Rust by default,
but that can interfere with our own compiler (for example Cargo's test
suite we think) so be sure to uninstall it before proceeding.
Speed up Azure CI installing Windows dependencies
There is known issue where PowerShell is unreasonably slow downloading
files due to an issue with rendering the progress bar, see this [issue](https://github.com/PowerShell/PowerShell/issues/2138)
That issue is fixed in PowerShell Core (available in Azure Pipelines as
pwsh.exe) but it can also be worked around by setting:
$ProgressPreference = 'SilentlyContinue'
I measured downloading LLVM and it took about 220s before, 5s after, so the improvement is significant.
ci: Favor SCRIPT instead of RUST_CHECK_TARGET
Since #61212 we've been timing out on OSX, and this looks to be because
we're building tools like Cargo and the RLS twice instead of once. This
turns out to be a slight bug in our configuration. CI builders using the
`RUST_CHECK_TARGET` directive actually execute `make all` just before
their acual target. In `make all` we're building a stage2 cargo, and
then in `make dist` we're building a stage1 cargo.
Other builders use `SCRIPT` which provides explicit control over what
`x.py` script, for example, is used to execute the build. This moves
almost all targets to using `SCRIPT` to ensure that we're explicitly
specifying what's being built where. Additionally this updates the logic
of `RUST_CHECK_TARGET` to remove the pre-flight tidy as well as the
pre-flight `make all`. The system LLVM builder (run on PRs) now
explicitly runs tidy first and then runs the rest of the test suite.
Since #61212 we've been timing out on OSX, and this looks to be because
we're building tools like Cargo and the RLS twice instead of once. This
turns out to be a slight bug in our configuration. CI builders using the
`RUST_CHECK_TARGET` directive actually execute `make all` just before
their acual target. In `make all` we're building a stage2 cargo, and
then in `make dist` we're building a stage1 cargo.
Other builders use `SCRIPT` which provides explicit control over what
`x.py` script, for example, is used to execute the build. This moves
almost all targets to using `SCRIPT` to ensure that we're explicitly
specifying what's being built where. Additionally this updates the logic
of `RUST_CHECK_TARGET` to remove the pre-flight tidy as well as the
pre-flight `make all`. The system LLVM builder (run on PRs) now
explicitly runs tidy first and then runs the rest of the test suite.
There is known issue where PowerShell is unreasonably slow downloading
files due to an issue with rendering the progress bar, see:
https://github.com/PowerShell/PowerShell/issues/2138
That issue is fixed in PowerShell Core (available in Azure Pipelines as
pwsh.exe) but it can also be worked around by setting:
$ProgressPreference = 'SilentlyContinue'
We were checking out the rustc repo multiple times on auto macOS, and
that was causing an error on the Azure side since multiple checkouts are
apparently not supported. This removes the extra checkout.
* Clean up the `install-windows-build-deps.yml` file and add some more
comments where appropriate.
* Add some comments to `run.yml`
* Don't fast path the `rustfmt` submodule, but we'll take care of that
later if necessary.
This commit is intended to go through and review/refactor the azure
pipelines configuration we have. The major changes are:
* The separate `{windows,macos,linux}.yml` files are now all merged into
one `run.yml`. This allows a shared "master flow" for all platforms
with divergence only where necessary.
* Some install steps have been separated as `install-*.yml` scripts,
where each script internally matches on the appropriate OS and then
delegates accordingly.
* Some various bits and pieces of cruft have been removed which were
artifacts of Travis's setup or similar.
