Commit Graph

168 Commits

Author SHA1 Message Date
Jakub Beránek
8e0a87bdf4
CI: use macos-13 runner for Apple jobs 2023-07-10 20:22:15 +02:00
Jakub Beránek
91d2fb2e2b
Port PGO/LTO/BOLT optimized build pipeline to Rust 2023-07-09 08:39:50 +02:00
Matthias Krüger
e0b290f1a8
Rollup merge of #113173 - Kobzol:ci-concurrency-group-workflow, r=pietroalbini
CI: include workflow name in concurrency group

Currently, this won't change anything, because we only have one relevant workflow (`CI`), but for future proofing we should probably include the workflow name in the concurrency group.

Found by ``@klensy`` [here](https://github.com/rust-lang/rust/pull/113059#discussion_r1247213606).
2023-07-08 15:49:45 +02:00
bors
06082086b4 Auto merge of #112796 - Kobzol:ci-merge-msvc-cargo-tools, r=pietroalbini
CI: merge msvc cargo and tools jobs

The `x86_64-msvc-cargo` and `x86_64-msvc-tools` jobs both run for ~1 hour, but most of that time is actually spent in building LLVM and `rustc`, so I want to try merging them.
![image](https://github.com/rust-lang/rust/assets/4539057/8652fa2a-b8b7-41d0-8f16-555d31acd9a5)
2023-07-07 00:39:47 +00:00
bors
87c8c83ec7 Auto merge of #112779 - Kobzol:ci-merge-llvm-14, r=pietroalbini
CI: merge x86_64-gnu-llvm-14 and x86_64-gnu-llvm-14-stage1 CI jobs

Another attempt to shorten CI job times. Suggested by `@the8472` [here](https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/CI.20usage/near/367172221).
2023-07-06 16:14:50 +00:00
Jubilee Young
634f95295f Remove MIPS host CI workflows 2023-07-02 22:04:46 -07:00
Jakub Beránek
f88024627b
CI: include workflow name in concurrency group
Currently, this won't change anything, because we only have one relevant workflow (`CI`), but for future proofing we should probably include the workflow name in the concurrency group.
2023-06-30 00:08:38 +02:00
Jakub Beránek
32428ab5f3
CI: do not cancel concurrent builds on the same branch
Add an exception for try and try-perf branches to enable concurrent try builds and unrolled rollup builds.
2023-06-29 14:52:52 +02:00
Jakub Beránek
7fb559353e
CI: do not cancel concurrent builds on the same branch
Instead, only cancel them if the builds have the same commit SHA.
2023-06-26 16:40:12 +02:00
Matthias Krüger
5d9935a638
Rollup merge of #112955 - Kobzol:ci-pr-cancel-workflows, r=Mark-Simulacrum
CI: cancel in-progress workflow runs after a push

Experimenting with the `concurrency` attribute.

r? `@Mark-Simulacrum`
2023-06-26 11:58:45 +02:00
Jakub Beránek
e8973eac12
Remove cancel-outdated action 2023-06-25 16:01:01 +02:00
Jakub Beránek
cb06c973c7
CI: do not run Bump dependencies workflow on forks 2023-06-25 15:55:27 +02:00
Jakub Beránek
3c2b8b06fc Cancel in-progress workflow runs after a push 2023-06-23 11:22:56 +02:00
Jakub Beránek
e115359035
CI: merge x86_64-msvc-cargo and x86_64-msvc-tools jobs 2023-06-19 15:58:12 +02:00
Jakub Beránek
1dba9721fc
CI: merge x86_64-gnu-llvm-14 and x86_64-gnu-llvm-14-stage1 CI jobs 2023-06-19 10:15:03 +02:00
bors
2d0aa57684 Auto merge of #112645 - Kobzol:ci-mingw-merge, r=pietroalbini
CI: merge `mingw` test CI jobs

Same as https://github.com/rust-lang/rust/pull/112633, but for `mingw`. From the logs it looks like the runner spends 40 minutes compiling `rustc`, and then `10`/`20` minutes running tests. It seems wasteful to split that into two jobs.

CI run: https://github.com/rust-lang/rust/actions/runs/5275702134/jobs/9541479343?pr=112645

r? `@jyn514`
2023-06-18 19:49:26 +00:00
Jakub Beránek
f3a4cf13a7
Merge mingw-1/2 CI jobs 2023-06-18 14:40:52 +02:00
Matthias Krüger
be0f3bdc3e
Rollup merge of #110805 - pitaj:master, r=Mark-Simulacrum
Github action to periodically `cargo update` to keep dependencies current

Opens a PR periodically with the results of `cargo update`. If an unmerged PR for the branch `cargo_update` already exists, it will edit then reopen it if necessary.

~~This also uses [`cargo-upgrades`](https://gitlab.com/kornelski/cargo-upgrades) to provide a list of available major upgrades in the PR body.~~

It includes the list of changes output by `cargo update` in the commit message and PR body. Note that this output is currently sub-optimal due to https://github.com/rust-lang/cargo/issues/9408, but if updates are made more regularly that is less likely to show up.

Example PR: https://github.com/pitaj/rust/pull/2
Example action run: https://github.com/pitaj/rust/actions/runs/5035731903
Prior discussion: https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/dependabot.20updates.3F

Up for discussion:
- What period do we want? Currently weekly
- What user should it use? Currently "Github Actions"
- Do we need the extra security of provided by executing `cargo update` and `cargo-upgrades` in a separate job?
  If not I can simplify it to not need artifacts.
- PR message wording
- PR should probably always be `rollup=always`?
- What branch should it use?
- What should it do if no updates are available? Currently fails the job on empty commit
- Should the yml file live in `src/ci` instead of directly under workflows?
- ~~Is using the latest nightly toolchain enough to ensure compatibility with `Cargo.lock` and `Cargo.toml`s in master?~~
  Now pulls the bootstrap version from stage0.json

r? infra
2023-06-17 18:27:30 +02:00
bors
1d7d824726 Auto merge of #112407 - tgross35:ci-docs-publish, r=Mark-Simulacrum
Publish docs as github artifacts during CI

Discussed here: https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/Building.20docs.20for.20PR.20CI

The goal is to make docs available for download after CI runs on PRs, for easy review of API changes.

Notes:

- Currently this only captures library documentation (`core`, `alloc`, `std`, `test`, `proc_macro`)
- You can't see artifacts until the entire workflow run has completed https://github.com/actions/upload-artifact/issues/53
- There is currently a generic file name `ci-artifacts`. No way to customize this based on contained files unfortunately https://github.com/actions/upload-artifact/issues/349

You can find the results at the bottom of the CI "summary" page:

<img width="379" alt="image" src="https://github.com/rust-lang/rust/assets/13724985/d3748e59-242c-40f8-9f54-82177b9b481b">
2023-06-17 03:28:53 +00:00
Jakub Beránek
895eb3035e
Merge msvc-1/2 CI jobs 2023-06-14 23:07:49 +02:00
Trevor Gross
696b0dd472 Publish docs as github artifacts during CI
This PR saves library docs as github artifacts so they can be easily
viewed for review.

Discussed in <https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/Building.20docs.20for.20PR.20CI>
2023-06-12 17:31:20 -04:00
Peter Jaszkowiak
6d3ff10536 run cargo update weekly
- Keep Cargo.lock dependencies current
- Presents output from `cargo update` in commit and PR
- Edit existing open PR, otherwise open a new one
- Skip if existing open PR is S-waiting-on-bors
2023-06-10 14:47:44 -06:00
Jakub Beránek
db113b1c2b
Do not build components unneeded for perf bot in try builds 2023-06-01 08:29:19 +02:00
WANG Rui
5f173e979f ci: Add support for dist-loongarch64-linux
Co-Authored-By: YANG Xiaojuan <yangxiaojuan@loongson.cn>
2023-05-12 18:31:55 +08:00
bors
c4190f2d3a Auto merge of #111224 - jyn514:default-tidy, r=pietroalbini
Remove `tidy` key in PR CI

This avoids confusing error messages when adding an `auto` job to CI (as recommended in the dev-guide: https://rustc-dev-guide.rust-lang.org/tests/ci.html#using-ci-to-test).

cc https://rust-lang.zulipchat.com/#narrow/stream/182449-t-compiler.2Fhelp/topic/Do.20.60needs-profiler-support.60.20tests.20not.20run.20in.20CI.3F/near/355302998

r? `@ghost`
2023-05-07 17:17:58 +00:00
bors
a77c552485 Auto merge of #110972 - ehuss:remove-awscli, r=pietroalbini
Remove aws cli install.

All runner images have the AWS CLI 2 installed, so there isn't a really strong reason to install our own version anymore.

The version we were installing was 1.27.122. The runner images currently have 2.11.x (the exact version varies by image).

I do not have the means to really test if the new version has any issues. I looked at all the `aws` commands, and none of them seem to be doing anything unusual. The page at https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration-changes.html contains a list of all the breaking changes, and I didn't see anything that looked important.
2023-05-06 21:15:25 +00:00
jyn
453c632b23 Remove tidy key in PR CI
This avoids confusing error messages when adding an `auto` job to CI (as
recommended in the dev-guide: https://rustc-dev-guide.rust-lang.org/tests/ci.html#using-ci-to-test).
2023-05-04 18:17:46 -05:00
bors
82cd953c7c Auto merge of #110846 - jdno:reduce-builder-sizes, r=pietroalbini
Optimize builder sizes

The infra-team is continuously monitoring the efficiency of the CI system in an effort to improve overall build times and resource usage. Some builders have used much less than their allocated resources, so we are testing smaller builder sizes for them.

r? `@pietroalbini`
2023-05-03 07:06:42 +00:00
Eric Huss
291b61e94f Set the AWS region. 2023-04-30 13:23:48 -07:00
Eric Huss
7cb798c24f Remove aws cli install. 2023-04-28 18:16:34 -07:00
Jan David
47528c0b14
Downsize builders for some x86_64-gnu targets
The infra-team is continuously monitoring the efficiency of the build
system in an effort to improve overall build times and resource usage.
The builders for some of the `x86_64-gnu` targets have used much less
resources than allocated in the past, so we are testing a smaller
builder size for them.
2023-04-26 16:52:39 +02:00
Jan David
4cbe65db8b
Downsize builder for mingw-check
The infra-team is continuously monitoring the efficiency of the build
system in an effort to improve overall build times and resource usage.
The builder for the `mingw-check` target have used much less resources
than allocated in the past, so we are testing a smaller builder size for
it.
2023-04-26 16:40:12 +02:00
Jan David
3402e286d5
Downsize builders for i686-gnu
The infra-team is continuously monitoring the efficiency of the build
system in an effort to improve overall build times and resource usage.
The builders for the `i686-gnu` targets have used much less resources
than allocated in the past, so we are testing a smaller builder size for
them.
2023-04-26 16:08:01 +02:00
Ryan Levick
241cbcaa87 Use the standard macOS CI runner 2023-04-25 10:32:29 +02:00
Josh Stone
33036159a4 ci: add a runner for vanilla LLVM 16
Like #107044, this will let us track compatibility with LLVM 16 going
forward, especially after we eventually upgrade our own to the next.

This also drops `tidy` here and in `x86_64-gnu-llvm-15`, syncing with
that change in #106085.
2023-04-16 11:50:20 -07:00
Mark Rousskov
3153eaaeb5 Trim down core counts for fast builders
These builders aren't particularly high on overall average CPU usage and finish in typically around
30 minutes. Cutting their core counts will hopefully not significantly increase wall-time while
cutting costs, allowing us to shift some of the wins into our slower builders.
2023-04-10 10:34:24 -04:00
jyn
423e76f1dd Improve job names in Github Actions preview
Before: `CI / PR (mingw-check, false, ubuntu-20.04-16core-64gb) (pull_request)`
After: `CI / PR - mingw-check (pull_request)`
2023-04-02 17:58:08 -04:00
Mark Rousskov
f83dfd9f86 Move us to the new large runners pool
For now this keeps all the configuration identical (AFAICT) but we'll
likely want to play with the specifics to move some of the slower
builders to larger machines and the faster builders to smaller machines,
likely reducing overall usage and improving CI times.
2023-03-19 11:41:55 -04:00
Nilstrieb
58884a30a0 Revert "enable ThinLTO for rustc on x86_64-pc-windows-msvc dist builds"
This lead to a miscompilation in at least `char::is_whitespace` and
probably in more unknown places.....

This reverts commit 684663ed38.
2023-03-13 21:18:54 +01:00
Michael Goulet
506ce7e3cc Re-apply "switch to the macos-12-xl builder"
This reverts commit e63ec2e140.
2023-02-22 22:14:25 +00:00
Josh Stone
a06aaa4a9e Update the minimum external LLVM to 14 2023-02-10 16:06:25 -08:00
Joshua Nelson
9dfec5d35b Run the tools builder on all PRs
Previously, it would only run on changes to subtrees, submodules, or select directories.
That made it so that changes to the compiler that broke tools would only be detected on a full bors merge.
This makes it so the tools builder runs by default, making it easier to catch breaking changes to clippy (which was the most effected).
2023-02-05 14:29:49 -06:00
bors
5d32458343 Auto merge of #107543 - ehuss:protocol-sparse, r=jyn514
Enable Cargo's sparse protocol in CI

This enables the sparse protocol in CI in order to exercise and dogfood it. This is intended test the production server in a real-world situation.

Closes #107342
2023-02-03 04:49:50 +00:00
Michael Goulet
e63ec2e140 Revert "switch to the macos-12-xl builder"
This reverts commit fcbae989ae.
2023-02-01 21:31:05 +00:00
Eric Huss
5e90940a4b Enable Cargo's sparse protocol in CI 2023-01-31 18:13:57 -08:00
bors
f55b0022db Auto merge of #103019 - Kobzol:ci-multistage-python, r=Mark-Simulacrum
Port pgo.sh to Python

This PR ports the `pgo.sh` multi stage build file from bash to Python, to make it easier to add new functionality and gather statistics. Main changes:

1) `pgo.sh` rewritten from Bash to Python. Jump from ~200 Bash LOC to ~650 Python LOC. Bash is, unsurprisingly, more concise for running scripts and binaries.
2) Better logging. Each separate stage is now clearly separated in logs, and the logs can be quickly grepped to find out which stage has completed or failed, and how long it took.
3) Better statistics. At the end of the run, there is now a table that shows the duration of the individual stages, along with a percentual ratio of the total workflow run:

```
2023-01-15T18:13:49.9896916Z stage-build INFO: Timer results
2023-01-15T18:13:49.9902185Z ---------------------------------------------------------
2023-01-15T18:13:49.9902605Z Build rustc (LLVM PGO):                 1815.67s (21.47%)
2023-01-15T18:13:49.9902949Z Gather profiles (LLVM PGO):              418.73s ( 4.95%)
2023-01-15T18:13:49.9903269Z Build rustc (rustc PGO):                 584.46s ( 6.91%)
2023-01-15T18:13:49.9903835Z Gather profiles (rustc PGO):             806.32s ( 9.53%)
2023-01-15T18:13:49.9904154Z Build rustc (LLVM BOLT):                1662.92s (19.66%)
2023-01-15T18:13:49.9904464Z Gather profiles (LLVM BOLT):             715.18s ( 8.46%)
2023-01-15T18:13:49.9914463Z Final build:                            2454.00s (29.02%)
2023-01-15T18:13:49.9914798Z Total duration:                         8457.27s
2023-01-15T18:13:49.9915305Z ---------------------------------------------------------
```

A sample run can be seen [here](https://github.com/rust-lang/rust/actions/runs/3923980164/jobs/6707932029).

I tried to keep the code compatible with Python 3.6 and don't use dependencies, which required me to reimplement some small pieces of functionality (like formatting bytes). I suppose that it shouldn't be so hard to upgrade to a newer Python or install dependencies in the CI container, but I'd like to avoid it if it won't be needed.

The code is in a single file `stage-build.py`, so it's a bit cluttered. I can also separate it into multiple files, although having it in a single file has some benefits. The code could definitely be nicer, but I'm a bit wary of introducing a lot of abstraction and similar stuff, as long as the code is stuffed into a single file.

Currently, the Python pipeline should faithfully mirror the bash pipeline one by one. After this PR, I'd like to try to optimize it, e.g. by caching the LLVM builds on S3.

r? `@Mark-Simulacrum`
2023-01-29 22:14:18 +00:00
Mateusz Mikuła
f6ea2ea551 Upgrade mingw-w64 on CI 2023-01-29 13:01:06 +01:00
Jakub Beránek
5b77cb459f
Port pgo.sh to Python 2023-01-24 21:51:01 +01:00
Josh Stone
318d77ddfd ci: add runners for vanilla LLVM 14 and 15 2023-01-18 11:14:40 -08:00
Albert Larsan
40ba0e84d5
Change src/test to tests in source files, fix tidy and tests 2023-01-11 09:32:13 +00:00