mikros/rust
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
rust/.github/workflows/ci.yml
bors f55b0022db Auto merge of #103019 - Kobzol:ci-multistage-python, r=Mark-Simulacrum 
Port pgo.sh to Python

This PR ports the `pgo.sh` multi stage build file from bash to Python, to make it easier to add new functionality and gather statistics. Main changes:

1) `pgo.sh` rewritten from Bash to Python. Jump from ~200 Bash LOC to ~650 Python LOC. Bash is, unsurprisingly, more concise for running scripts and binaries.
2) Better logging. Each separate stage is now clearly separated in logs, and the logs can be quickly grepped to find out which stage has completed or failed, and how long it took.
3) Better statistics. At the end of the run, there is now a table that shows the duration of the individual stages, along with a percentual ratio of the total workflow run:

```
2023-01-15T18:13:49.9896916Z stage-build INFO: Timer results
2023-01-15T18:13:49.9902185Z ---------------------------------------------------------
2023-01-15T18:13:49.9902605Z Build rustc (LLVM PGO):                 1815.67s (21.47%)
2023-01-15T18:13:49.9902949Z Gather profiles (LLVM PGO):              418.73s ( 4.95%)
2023-01-15T18:13:49.9903269Z Build rustc (rustc PGO):                 584.46s ( 6.91%)
2023-01-15T18:13:49.9903835Z Gather profiles (rustc PGO):             806.32s ( 9.53%)
2023-01-15T18:13:49.9904154Z Build rustc (LLVM BOLT):                1662.92s (19.66%)
2023-01-15T18:13:49.9904464Z Gather profiles (LLVM BOLT):             715.18s ( 8.46%)
2023-01-15T18:13:49.9914463Z Final build:                            2454.00s (29.02%)
2023-01-15T18:13:49.9914798Z Total duration:                         8457.27s
2023-01-15T18:13:49.9915305Z ---------------------------------------------------------
```

A sample run can be seen [here](https://github.com/rust-lang/rust/actions/runs/3923980164/jobs/6707932029).

I tried to keep the code compatible with Python 3.6 and don't use dependencies, which required me to reimplement some small pieces of functionality (like formatting bytes). I suppose that it shouldn't be so hard to upgrade to a newer Python or install dependencies in the CI container, but I'd like to avoid it if it won't be needed.

The code is in a single file `stage-build.py`, so it's a bit cluttered. I can also separate it into multiple files, although having it in a single file has some benefits. The code could definitely be nicer, but I'm a bit wary of introducing a lot of abstraction and similar stuff, as long as the code is stuffed into a single file.

Currently, the Python pipeline should faithfully mirror the bash pipeline one by one. After this PR, I'd like to try to optimize it, e.g. by caching the LLVM builds on S3.

r? `@Mark-Simulacrum`
2023-01-29 22:14:18 +00:00

760 lines
30 KiB
YAML
Raw Blame History

#############################################################
# WARNING: automatically generated file, DO NOT CHANGE! #
#############################################################
# This file was automatically generated by the expand-yaml-anchors tool. The
# source file that generated this one is:
#
# src/ci/github-actions/ci.yml
#
# Once you make changes to that file you need to run:
#
# ./x.py run src/tools/expand-yaml-anchors/
#
# The CI build will fail if the tool is not run after changes to this file.
---
name: CI
"on":
push:
branches:
- auto
- try
- try-perf
- master
pull_request:
branches:
- "**"
permissions:
contents: read
defaults:
run:
shell: bash
jobs:
pr:
permissions:
actions: write
name: PR
env:
CI_JOB_NAME: "${{ matrix.name }}"
SCCACHE_BUCKET: rust-lang-ci-sccache2
TOOLSTATE_REPO: "https://github.com/rust-lang-nursery/rust-toolstate"
CACHE_DOMAIN: ci-caches.rust-lang.org
if: "github.event_name == 'pull_request'"
continue-on-error: "${{ matrix.tidy }}"
strategy:
matrix:
include:
- name: mingw-check
tidy: false
os: ubuntu-20.04-xl
env: {}
- name: mingw-check-tidy
tidy: true
os: ubuntu-20.04-xl
env: {}
- name: x86_64-gnu-llvm-13
tidy: false
os: ubuntu-20.04-xl
env: {}
- name: x86_64-gnu-tools
tidy: false
env:
CI_ONLY_WHEN_SUBMODULES_CHANGED: 1
os: ubuntu-20.04-xl
timeout-minutes: 600
runs-on: "${{ matrix.os }}"
steps:
- name: disable git crlf conversion
run: git config --global core.autocrlf false
- name: checkout the source code
uses: actions/checkout@v3
with:
fetch-depth: 2
- name: configure the PR in which the error message will be posted
run: "echo \"[CI_PR_NUMBER=$num]\""
env:
num: "${{ github.event.number }}"
if: "success() && !env.SKIP_JOB && github.event_name == 'pull_request'"
- name: add extra environment variables
run: src/ci/scripts/setup-environment.sh
env:
EXTRA_VARIABLES: "${{ toJson(matrix.env) }}"
if: success() && !env.SKIP_JOB
- name: decide whether to skip this job
run: src/ci/scripts/should-skip-this.sh
if: success() && !env.SKIP_JOB
- name: ensure the channel matches the target branch
run: src/ci/scripts/verify-channel.sh
if: success() && !env.SKIP_JOB
- name: configure GitHub Actions to kill the build when outdated
uses: rust-lang/simpleinfra/github-actions/cancel-outdated-builds@master
with:
github_token: "${{ secrets.github_token }}"
if: "success() && !env.SKIP_JOB && github.ref != 'refs/heads/try' && github.ref != 'refs/heads/try-perf'"
- name: collect CPU statistics
run: src/ci/scripts/collect-cpu-stats.sh
if: success() && !env.SKIP_JOB
- name: show the current environment
run: src/ci/scripts/dump-environment.sh
if: success() && !env.SKIP_JOB
- name: install awscli
run: src/ci/scripts/install-awscli.sh
if: success() && !env.SKIP_JOB
- name: install sccache
run: src/ci/scripts/install-sccache.sh
if: success() && !env.SKIP_JOB
- name: select Xcode
run: src/ci/scripts/select-xcode.sh
if: success() && !env.SKIP_JOB
- name: install clang
run: src/ci/scripts/install-clang.sh
if: success() && !env.SKIP_JOB
- name: install WIX
run: src/ci/scripts/install-wix.sh
if: success() && !env.SKIP_JOB
- name: disable git crlf conversion
run: src/ci/scripts/disable-git-crlf-conversion.sh
if: success() && !env.SKIP_JOB
- name: checkout submodules
run: src/ci/scripts/checkout-submodules.sh
if: success() && !env.SKIP_JOB
- name: install MSYS2
run: src/ci/scripts/install-msys2.sh
if: success() && !env.SKIP_JOB
- name: install MinGW
run: src/ci/scripts/install-mingw.sh
if: success() && !env.SKIP_JOB
- name: install ninja
run: src/ci/scripts/install-ninja.sh
if: success() && !env.SKIP_JOB
- name: enable ipv6 on Docker
run: src/ci/scripts/enable-docker-ipv6.sh
if: success() && !env.SKIP_JOB
- name: disable git crlf conversion
run: src/ci/scripts/disable-git-crlf-conversion.sh
if: success() && !env.SKIP_JOB
- name: ensure line endings are correct
run: src/ci/scripts/verify-line-endings.sh
if: success() && !env.SKIP_JOB
- name: ensure backported commits are in upstream branches
run: src/ci/scripts/verify-backported-commits.sh
if: success() && !env.SKIP_JOB
- name: ensure the stable version number is correct
run: src/ci/scripts/verify-stable-version-number.sh
if: success() && !env.SKIP_JOB
- name: run the build
run: src/ci/scripts/run-build-from-ci.sh
env:
AWS_ACCESS_KEY_ID: "${{ env.CACHES_AWS_ACCESS_KEY_ID }}"
AWS_SECRET_ACCESS_KEY: "${{ secrets[format('AWS_SECRET_ACCESS_KEY_{0}', env.CACHES_AWS_ACCESS_KEY_ID)] }}"
TOOLSTATE_REPO_ACCESS_TOKEN: "${{ secrets.TOOLSTATE_REPO_ACCESS_TOKEN }}"
if: success() && !env.SKIP_JOB
- name: upload artifacts to S3
run: src/ci/scripts/upload-artifacts.sh
env:
AWS_ACCESS_KEY_ID: "${{ env.ARTIFACTS_AWS_ACCESS_KEY_ID }}"
AWS_SECRET_ACCESS_KEY: "${{ secrets[format('AWS_SECRET_ACCESS_KEY_{0}', env.ARTIFACTS_AWS_ACCESS_KEY_ID)] }}"
if: "success() && !env.SKIP_JOB && (github.event_name == 'push' || env.DEPLOY == '1' || env.DEPLOY_ALT == '1')"
auto:
permissions:
actions: write
name: auto
env:
CI_JOB_NAME: "${{ matrix.name }}"
SCCACHE_BUCKET: rust-lang-ci-sccache2
DEPLOY_BUCKET: rust-lang-ci2
TOOLSTATE_REPO: "https://github.com/rust-lang-nursery/rust-toolstate"
TOOLSTATE_ISSUES_API_URL: "https://api.github.com/repos/rust-lang/rust/issues"
TOOLSTATE_PUBLISH: 1
CACHES_AWS_ACCESS_KEY_ID: AKIA46X5W6CZI5DHEBFL
ARTIFACTS_AWS_ACCESS_KEY_ID: AKIA46X5W6CZN24CBO55
CACHE_DOMAIN: ci-caches.rust-lang.org
if: "github.event_name == 'push' && github.ref == 'refs/heads/auto' && github.repository == 'rust-lang-ci/rust'"
strategy:
matrix:
include:
- name: aarch64-gnu
os:
- self-hosted
- ARM64
- linux
- name: arm-android
os: ubuntu-20.04-xl
env: {}
- name: armhf-gnu
os: ubuntu-20.04-xl
env: {}
- name: dist-aarch64-linux
os: ubuntu-20.04-xl
env: {}
- name: dist-android
os: ubuntu-20.04-xl
env: {}
- name: dist-arm-linux
os: ubuntu-20.04-xl
env: {}
- name: dist-armhf-linux
os: ubuntu-20.04-xl
env: {}
- name: dist-armv7-linux
os: ubuntu-20.04-xl
env: {}
- name: dist-i586-gnu-i586-i686-musl
os: ubuntu-20.04-xl
env: {}
- name: dist-i686-linux
os: ubuntu-20.04-xl
env: {}
- name: dist-mips-linux
os: ubuntu-20.04-xl
env: {}
- name: dist-mips64-linux
os: ubuntu-20.04-xl
env: {}
- name: dist-mips64el-linux
os: ubuntu-20.04-xl
env: {}
- name: dist-mipsel-linux
os: ubuntu-20.04-xl
env: {}
- name: dist-powerpc-linux
os: ubuntu-20.04-xl
env: {}
- name: dist-powerpc64-linux
os: ubuntu-20.04-xl
env: {}
- name: dist-powerpc64le-linux
os: ubuntu-20.04-xl
env: {}
- name: dist-riscv64-linux
os: ubuntu-20.04-xl
env: {}
- name: dist-s390x-linux
os: ubuntu-20.04-xl
env: {}
- name: dist-various-1
os: ubuntu-20.04-xl
env: {}
- name: dist-various-2
os: ubuntu-20.04-xl
env: {}
- name: dist-x86_64-freebsd
os: ubuntu-20.04-xl
env: {}
- name: dist-x86_64-illumos
os: ubuntu-20.04-xl
env: {}
- name: dist-x86_64-linux
os: ubuntu-20.04-xl
env: {}
- name: dist-x86_64-linux-alt
env:
IMAGE: dist-x86_64-linux
os: ubuntu-20.04-xl
- name: dist-x86_64-musl
os: ubuntu-20.04-xl
env: {}
- name: dist-x86_64-netbsd
os: ubuntu-20.04-xl
env: {}
- name: i686-gnu
os: ubuntu-20.04-xl
env: {}
- name: i686-gnu-nopt
os: ubuntu-20.04-xl
env: {}
- name: mingw-check
os: ubuntu-20.04-xl
env: {}
- name: test-various
os: ubuntu-20.04-xl
env: {}
- name: wasm32
os: ubuntu-20.04-xl
env: {}
- name: x86_64-gnu
os: ubuntu-20.04-xl
env: {}
- name: x86_64-gnu-stable
env:
IMAGE: x86_64-gnu
RUST_CI_OVERRIDE_RELEASE_CHANNEL: stable
CI_ONLY_WHEN_CHANNEL: nightly
os: ubuntu-20.04-xl
- name: x86_64-gnu-aux
os: ubuntu-20.04-xl
env: {}
- name: x86_64-gnu-debug
os: ubuntu-20.04-xl
env: {}
- name: x86_64-gnu-distcheck
os: ubuntu-20.04-xl
env: {}
- name: x86_64-gnu-llvm-15
env:
RUST_BACKTRACE: 1
os: ubuntu-20.04-xl
- name: x86_64-gnu-llvm-14
env:
RUST_BACKTRACE: 1
os: ubuntu-20.04-xl
- name: x86_64-gnu-llvm-13
env:
RUST_BACKTRACE: 1
os: ubuntu-20.04-xl
- name: x86_64-gnu-llvm-13-stage1
env:
RUST_BACKTRACE: 1
os: ubuntu-20.04-xl
- name: x86_64-gnu-nopt
os: ubuntu-20.04-xl
env: {}
- name: x86_64-gnu-tools
env:
DEPLOY_TOOLSTATES_JSON: toolstates-linux.json
os: ubuntu-20.04-xl
- name: dist-x86_64-apple
env:
SCRIPT: "./x.py dist bootstrap --include-default-paths --host=x86_64-apple-darwin --target=x86_64-apple-darwin"
RUST_CONFIGURE_ARGS: "--enable-full-tools --enable-sanitizers --enable-profiler --set rust.jemalloc --set llvm.ninja=false --set rust.lto=thin"
RUSTC_RETRY_LINKER_ON_SEGFAULT: 1
MACOSX_DEPLOYMENT_TARGET: 10.7
SELECT_XCODE: /Applications/Xcode_13.4.1.app
NO_LLVM_ASSERTIONS: 1
NO_DEBUG_ASSERTIONS: 1
NO_OVERFLOW_CHECKS: 1
DIST_REQUIRE_ALL_TOOLS: 1
os: macos-12-xl
- name: dist-apple-various
env:
SCRIPT: "./x.py dist bootstrap --include-default-paths --host='' --target=aarch64-apple-ios,x86_64-apple-ios,aarch64-apple-ios-sim"
RUST_CONFIGURE_ARGS: "--enable-sanitizers --enable-profiler --set rust.jemalloc --set llvm.ninja=false"
RUSTC_RETRY_LINKER_ON_SEGFAULT: 1
MACOSX_DEPLOYMENT_TARGET: 10.7
SELECT_XCODE: /Applications/Xcode_13.4.1.app
NO_LLVM_ASSERTIONS: 1
NO_DEBUG_ASSERTIONS: 1
NO_OVERFLOW_CHECKS: 1
os: macos-12-xl
- name: dist-x86_64-apple-alt
env:
SCRIPT: "./x.py dist bootstrap --include-default-paths"
RUST_CONFIGURE_ARGS: "--enable-extended --enable-profiler --set rust.jemalloc --set llvm.ninja=false"
RUSTC_RETRY_LINKER_ON_SEGFAULT: 1
MACOSX_DEPLOYMENT_TARGET: 10.7
SELECT_XCODE: /Applications/Xcode_13.4.1.app
NO_LLVM_ASSERTIONS: 1
NO_DEBUG_ASSERTIONS: 1
NO_OVERFLOW_CHECKS: 1
os: macos-12-xl
- name: x86_64-apple-1
env:
SCRIPT: "./x.py --stage 2 test --exclude tests/ui --exclude tests/rustdoc --exclude tests/run-make-fulldeps"
RUST_CONFIGURE_ARGS: "--build=x86_64-apple-darwin --enable-sanitizers --enable-profiler --set rust.jemalloc --set llvm.ninja=false"
RUSTC_RETRY_LINKER_ON_SEGFAULT: 1
MACOSX_DEPLOYMENT_TARGET: 10.8
MACOSX_STD_DEPLOYMENT_TARGET: 10.7
NO_LLVM_ASSERTIONS: 1
NO_DEBUG_ASSERTIONS: 1
NO_OVERFLOW_CHECKS: 1
os: macos-12-xl
- name: x86_64-apple-2
env:
SCRIPT: "./x.py --stage 2 test tests/ui tests/rustdoc tests/run-make-fulldeps"
RUST_CONFIGURE_ARGS: "--build=x86_64-apple-darwin --enable-sanitizers --enable-profiler --set rust.jemalloc --set llvm.ninja=false"
RUSTC_RETRY_LINKER_ON_SEGFAULT: 1
MACOSX_DEPLOYMENT_TARGET: 10.8
MACOSX_STD_DEPLOYMENT_TARGET: 10.7
NO_LLVM_ASSERTIONS: 1
NO_DEBUG_ASSERTIONS: 1
NO_OVERFLOW_CHECKS: 1
os: macos-12-xl
- name: dist-aarch64-apple
env:
SCRIPT: "./x.py dist bootstrap --include-default-paths --stage 2"
RUST_CONFIGURE_ARGS: "--build=x86_64-apple-darwin --host=aarch64-apple-darwin --target=aarch64-apple-darwin --enable-full-tools --enable-sanitizers --enable-profiler --disable-docs --set rust.jemalloc --set llvm.ninja=false"
RUSTC_RETRY_LINKER_ON_SEGFAULT: 1
SELECT_XCODE: /Applications/Xcode_13.4.1.app
USE_XCODE_CLANG: 1
MACOSX_DEPLOYMENT_TARGET: 11.0
MACOSX_STD_DEPLOYMENT_TARGET: 11.0
NO_LLVM_ASSERTIONS: 1
NO_DEBUG_ASSERTIONS: 1
NO_OVERFLOW_CHECKS: 1
DIST_REQUIRE_ALL_TOOLS: 1
JEMALLOC_SYS_WITH_LG_PAGE: 14
os: macos-12-xl
- name: x86_64-msvc-1
env:
RUST_CONFIGURE_ARGS: "--build=x86_64-pc-windows-msvc --enable-profiler"
SCRIPT: make ci-subset-1
os: windows-latest-xl
- name: x86_64-msvc-2
env:
RUST_CONFIGURE_ARGS: "--build=x86_64-pc-windows-msvc --enable-profiler"
SCRIPT: make ci-subset-2
os: windows-latest-xl
- name: i686-msvc-1
env:
RUST_CONFIGURE_ARGS: "--build=i686-pc-windows-msvc"
SCRIPT: make ci-subset-1
os: windows-latest-xl
- name: i686-msvc-2
env:
RUST_CONFIGURE_ARGS: "--build=i686-pc-windows-msvc"
SCRIPT: make ci-subset-2
os: windows-latest-xl
- name: x86_64-msvc-cargo
env:
SCRIPT: python x.py --stage 2 test src/tools/cargotest src/tools/cargo
RUST_CONFIGURE_ARGS: "--build=x86_64-pc-windows-msvc --enable-lld"
os: windows-latest-xl
- name: x86_64-msvc-tools
env:
SCRIPT: src/ci/docker/host-x86_64/x86_64-gnu-tools/checktools.sh x.py /tmp/toolstate/toolstates.json windows
RUST_CONFIGURE_ARGS: "--build=x86_64-pc-windows-msvc --save-toolstates=/tmp/toolstate/toolstates.json"
DEPLOY_TOOLSTATES_JSON: toolstates-windows.json
os: windows-latest-xl
- name: i686-mingw-1
env:
RUST_CONFIGURE_ARGS: "--build=i686-pc-windows-gnu"
SCRIPT: make ci-mingw-subset-1
NO_DOWNLOAD_CI_LLVM: 1
CUSTOM_MINGW: 1
os: windows-latest-xl
- name: i686-mingw-2
env:
RUST_CONFIGURE_ARGS: "--build=i686-pc-windows-gnu"
SCRIPT: make ci-mingw-subset-2
NO_DOWNLOAD_CI_LLVM: 1
CUSTOM_MINGW: 1
os: windows-latest-xl
- name: x86_64-mingw-1
env:
SCRIPT: make ci-mingw-subset-1
RUST_CONFIGURE_ARGS: "--build=x86_64-pc-windows-gnu --enable-profiler"
NO_DOWNLOAD_CI_LLVM: 1
CUSTOM_MINGW: 1
os: windows-latest-xl
- name: x86_64-mingw-2
env:
SCRIPT: make ci-mingw-subset-2
RUST_CONFIGURE_ARGS: "--build=x86_64-pc-windows-gnu --enable-profiler"
NO_DOWNLOAD_CI_LLVM: 1
CUSTOM_MINGW: 1
os: windows-latest-xl
- name: dist-x86_64-msvc
env:
RUST_CONFIGURE_ARGS: "--build=x86_64-pc-windows-msvc --host=x86_64-pc-windows-msvc --target=x86_64-pc-windows-msvc --enable-full-tools --enable-profiler --set rust.lto=thin"
SCRIPT: PGO_HOST=x86_64-pc-windows-msvc python src/ci/stage-build.py python x.py dist bootstrap --include-default-paths
DIST_REQUIRE_ALL_TOOLS: 1
os: windows-latest-xl
- name: dist-i686-msvc
env:
RUST_CONFIGURE_ARGS: "--build=i686-pc-windows-msvc --host=i686-pc-windows-msvc --target=i686-pc-windows-msvc,i586-pc-windows-msvc --enable-full-tools --enable-profiler"
SCRIPT: python x.py dist bootstrap --include-default-paths
DIST_REQUIRE_ALL_TOOLS: 1
os: windows-latest-xl
- name: dist-aarch64-msvc
env:
RUST_CONFIGURE_ARGS: "--build=x86_64-pc-windows-msvc --host=aarch64-pc-windows-msvc --enable-full-tools --enable-profiler"
SCRIPT: python x.py dist bootstrap --include-default-paths
DIST_REQUIRE_ALL_TOOLS: 1
WINDOWS_SDK_20348_HACK: 1
os: windows-latest-xl
- name: dist-i686-mingw
env:
RUST_CONFIGURE_ARGS: "--build=i686-pc-windows-gnu --enable-full-tools --enable-profiler"
NO_DOWNLOAD_CI_LLVM: 1
SCRIPT: python x.py dist bootstrap --include-default-paths
CUSTOM_MINGW: 1
DIST_REQUIRE_ALL_TOOLS: 1
os: windows-latest-xl
- name: dist-x86_64-mingw
env:
SCRIPT: python x.py dist bootstrap --include-default-paths
RUST_CONFIGURE_ARGS: "--build=x86_64-pc-windows-gnu --enable-full-tools --enable-profiler"
NO_DOWNLOAD_CI_LLVM: 1
CUSTOM_MINGW: 1
DIST_REQUIRE_ALL_TOOLS: 1
os: windows-latest-xl
- name: dist-x86_64-msvc-alt
env:
RUST_CONFIGURE_ARGS: "--build=x86_64-pc-windows-msvc --enable-extended --enable-profiler"
SCRIPT: python x.py dist bootstrap --include-default-paths
os: windows-latest-xl
timeout-minutes: 600
runs-on: "${{ matrix.os }}"
steps:
- name: disable git crlf conversion
run: git config --global core.autocrlf false
- name: checkout the source code
uses: actions/checkout@v3
with:
fetch-depth: 2
- name: configure the PR in which the error message will be posted
run: "echo \"[CI_PR_NUMBER=$num]\""
env:
num: "${{ github.event.number }}"
if: "success() && !env.SKIP_JOB && github.event_name == 'pull_request'"
- name: add extra environment variables
run: src/ci/scripts/setup-environment.sh
env:
EXTRA_VARIABLES: "${{ toJson(matrix.env) }}"
if: success() && !env.SKIP_JOB
- name: decide whether to skip this job
run: src/ci/scripts/should-skip-this.sh
if: success() && !env.SKIP_JOB
- name: ensure the channel matches the target branch
run: src/ci/scripts/verify-channel.sh
if: success() && !env.SKIP_JOB
- name: configure GitHub Actions to kill the build when outdated
uses: rust-lang/simpleinfra/github-actions/cancel-outdated-builds@master
with:
github_token: "${{ secrets.github_token }}"
if: "success() && !env.SKIP_JOB && github.ref != 'refs/heads/try' && github.ref != 'refs/heads/try-perf'"
- name: collect CPU statistics
run: src/ci/scripts/collect-cpu-stats.sh
if: success() && !env.SKIP_JOB
- name: show the current environment
run: src/ci/scripts/dump-environment.sh
if: success() && !env.SKIP_JOB
- name: install awscli
run: src/ci/scripts/install-awscli.sh
if: success() && !env.SKIP_JOB
- name: install sccache
run: src/ci/scripts/install-sccache.sh
if: success() && !env.SKIP_JOB
- name: select Xcode
run: src/ci/scripts/select-xcode.sh
if: success() && !env.SKIP_JOB
- name: install clang
run: src/ci/scripts/install-clang.sh
if: success() && !env.SKIP_JOB
- name: install WIX
run: src/ci/scripts/install-wix.sh
if: success() && !env.SKIP_JOB
- name: disable git crlf conversion
run: src/ci/scripts/disable-git-crlf-conversion.sh
if: success() && !env.SKIP_JOB
- name: checkout submodules
run: src/ci/scripts/checkout-submodules.sh
if: success() && !env.SKIP_JOB
- name: install MSYS2
run: src/ci/scripts/install-msys2.sh
if: success() && !env.SKIP_JOB
- name: install MinGW
run: src/ci/scripts/install-mingw.sh
if: success() && !env.SKIP_JOB
- name: install ninja
run: src/ci/scripts/install-ninja.sh
if: success() && !env.SKIP_JOB
- name: enable ipv6 on Docker
run: src/ci/scripts/enable-docker-ipv6.sh
if: success() && !env.SKIP_JOB
- name: disable git crlf conversion
run: src/ci/scripts/disable-git-crlf-conversion.sh
if: success() && !env.SKIP_JOB
- name: ensure line endings are correct
run: src/ci/scripts/verify-line-endings.sh
if: success() && !env.SKIP_JOB
- name: ensure backported commits are in upstream branches
run: src/ci/scripts/verify-backported-commits.sh
if: success() && !env.SKIP_JOB
- name: ensure the stable version number is correct
run: src/ci/scripts/verify-stable-version-number.sh
if: success() && !env.SKIP_JOB
- name: run the build
run: src/ci/scripts/run-build-from-ci.sh
env:
AWS_ACCESS_KEY_ID: "${{ env.CACHES_AWS_ACCESS_KEY_ID }}"
AWS_SECRET_ACCESS_KEY: "${{ secrets[format('AWS_SECRET_ACCESS_KEY_{0}', env.CACHES_AWS_ACCESS_KEY_ID)] }}"
TOOLSTATE_REPO_ACCESS_TOKEN: "${{ secrets.TOOLSTATE_REPO_ACCESS_TOKEN }}"
if: success() && !env.SKIP_JOB
- name: upload artifacts to S3
run: src/ci/scripts/upload-artifacts.sh
env:
AWS_ACCESS_KEY_ID: "${{ env.ARTIFACTS_AWS_ACCESS_KEY_ID }}"
AWS_SECRET_ACCESS_KEY: "${{ secrets[format('AWS_SECRET_ACCESS_KEY_{0}', env.ARTIFACTS_AWS_ACCESS_KEY_ID)] }}"
if: "success() && !env.SKIP_JOB && (github.event_name == 'push' || env.DEPLOY == '1' || env.DEPLOY_ALT == '1')"
try:
permissions:
actions: write
name: try
env:
CI_JOB_NAME: "${{ matrix.name }}"
SCCACHE_BUCKET: rust-lang-ci-sccache2
DEPLOY_BUCKET: rust-lang-ci2
TOOLSTATE_REPO: "https://github.com/rust-lang-nursery/rust-toolstate"
TOOLSTATE_ISSUES_API_URL: "https://api.github.com/repos/rust-lang/rust/issues"
TOOLSTATE_PUBLISH: 1
CACHES_AWS_ACCESS_KEY_ID: AKIA46X5W6CZI5DHEBFL
ARTIFACTS_AWS_ACCESS_KEY_ID: AKIA46X5W6CZN24CBO55
CACHE_DOMAIN: ci-caches.rust-lang.org
if: "github.event_name == 'push' && (github.ref == 'refs/heads/try' || github.ref == 'refs/heads/try-perf') && github.repository == 'rust-lang-ci/rust'"
strategy:
matrix:
include:
- name: dist-x86_64-linux
os: ubuntu-20.04-xl
env: {}
timeout-minutes: 600
runs-on: "${{ matrix.os }}"
steps:
- name: disable git crlf conversion
run: git config --global core.autocrlf false
- name: checkout the source code
uses: actions/checkout@v3
with:
fetch-depth: 2
- name: configure the PR in which the error message will be posted
run: "echo \"[CI_PR_NUMBER=$num]\""
env:
num: "${{ github.event.number }}"
if: "success() && !env.SKIP_JOB && github.event_name == 'pull_request'"
- name: add extra environment variables
run: src/ci/scripts/setup-environment.sh
env:
EXTRA_VARIABLES: "${{ toJson(matrix.env) }}"
if: success() && !env.SKIP_JOB
- name: decide whether to skip this job
run: src/ci/scripts/should-skip-this.sh
if: success() && !env.SKIP_JOB
- name: ensure the channel matches the target branch
run: src/ci/scripts/verify-channel.sh
if: success() && !env.SKIP_JOB
- name: configure GitHub Actions to kill the build when outdated
uses: rust-lang/simpleinfra/github-actions/cancel-outdated-builds@master
with:
github_token: "${{ secrets.github_token }}"
if: "success() && !env.SKIP_JOB && github.ref != 'refs/heads/try' && github.ref != 'refs/heads/try-perf'"
- name: collect CPU statistics
run: src/ci/scripts/collect-cpu-stats.sh
if: success() && !env.SKIP_JOB
- name: show the current environment
run: src/ci/scripts/dump-environment.sh
if: success() && !env.SKIP_JOB
- name: install awscli
run: src/ci/scripts/install-awscli.sh
if: success() && !env.SKIP_JOB
- name: install sccache
run: src/ci/scripts/install-sccache.sh
if: success() && !env.SKIP_JOB
- name: select Xcode
run: src/ci/scripts/select-xcode.sh
if: success() && !env.SKIP_JOB
- name: install clang
run: src/ci/scripts/install-clang.sh
if: success() && !env.SKIP_JOB
- name: install WIX
run: src/ci/scripts/install-wix.sh
if: success() && !env.SKIP_JOB
- name: disable git crlf conversion
run: src/ci/scripts/disable-git-crlf-conversion.sh
if: success() && !env.SKIP_JOB
- name: checkout submodules
run: src/ci/scripts/checkout-submodules.sh
if: success() && !env.SKIP_JOB
- name: install MSYS2
run: src/ci/scripts/install-msys2.sh
if: success() && !env.SKIP_JOB
- name: install MinGW
run: src/ci/scripts/install-mingw.sh
if: success() && !env.SKIP_JOB
- name: install ninja
run: src/ci/scripts/install-ninja.sh
if: success() && !env.SKIP_JOB
- name: enable ipv6 on Docker
run: src/ci/scripts/enable-docker-ipv6.sh
if: success() && !env.SKIP_JOB
- name: disable git crlf conversion
run: src/ci/scripts/disable-git-crlf-conversion.sh
if: success() && !env.SKIP_JOB
- name: ensure line endings are correct
run: src/ci/scripts/verify-line-endings.sh
if: success() && !env.SKIP_JOB
- name: ensure backported commits are in upstream branches
run: src/ci/scripts/verify-backported-commits.sh
if: success() && !env.SKIP_JOB
- name: ensure the stable version number is correct
run: src/ci/scripts/verify-stable-version-number.sh
if: success() && !env.SKIP_JOB
- name: run the build
run: src/ci/scripts/run-build-from-ci.sh
env:
AWS_ACCESS_KEY_ID: "${{ env.CACHES_AWS_ACCESS_KEY_ID }}"
AWS_SECRET_ACCESS_KEY: "${{ secrets[format('AWS_SECRET_ACCESS_KEY_{0}', env.CACHES_AWS_ACCESS_KEY_ID)] }}"
TOOLSTATE_REPO_ACCESS_TOKEN: "${{ secrets.TOOLSTATE_REPO_ACCESS_TOKEN }}"
if: success() && !env.SKIP_JOB
- name: upload artifacts to S3
run: src/ci/scripts/upload-artifacts.sh
env:
AWS_ACCESS_KEY_ID: "${{ env.ARTIFACTS_AWS_ACCESS_KEY_ID }}"
AWS_SECRET_ACCESS_KEY: "${{ secrets[format('AWS_SECRET_ACCESS_KEY_{0}', env.ARTIFACTS_AWS_ACCESS_KEY_ID)] }}"
if: "success() && !env.SKIP_JOB && (github.event_name == 'push' || env.DEPLOY == '1' || env.DEPLOY_ALT == '1')"
master:
name: master
runs-on: ubuntu-latest
env:
SCCACHE_BUCKET: rust-lang-ci-sccache2
DEPLOY_BUCKET: rust-lang-ci2
TOOLSTATE_REPO: "https://github.com/rust-lang-nursery/rust-toolstate"
TOOLSTATE_ISSUES_API_URL: "https://api.github.com/repos/rust-lang/rust/issues"
TOOLSTATE_PUBLISH: 1
CACHES_AWS_ACCESS_KEY_ID: AKIA46X5W6CZI5DHEBFL
ARTIFACTS_AWS_ACCESS_KEY_ID: AKIA46X5W6CZN24CBO55
CACHE_DOMAIN: ci-caches.rust-lang.org
if: "github.event_name == 'push' && github.ref == 'refs/heads/master' && github.repository == 'rust-lang-ci/rust'"
steps:
- name: checkout the source code
uses: actions/checkout@v3
with:
fetch-depth: 2
- name: publish toolstate
run: src/ci/publish_toolstate.sh
shell: bash
env:
TOOLSTATE_REPO_ACCESS_TOKEN: "${{ secrets.TOOLSTATE_REPO_ACCESS_TOKEN }}"
if: success() && !env.SKIP_JOB
try-success:
needs:
- try
if: "success() && github.event_name == 'push' && (github.ref == 'refs/heads/try' || github.ref == 'refs/heads/try-perf') && github.repository == 'rust-lang-ci/rust'"
steps:
- name: mark the job as a success
run: exit 0
shell: bash
name: bors build finished
runs-on: ubuntu-latest
try-failure:
needs:
- try
if: "!success() && github.event_name == 'push' && (github.ref == 'refs/heads/try' || github.ref == 'refs/heads/try-perf') && github.repository == 'rust-lang-ci/rust'"
steps:
- name: mark the job as a failure
run: exit 1
shell: bash
name: bors build finished
runs-on: ubuntu-latest
auto-success:
needs:
- auto
if: "success() && github.event_name == 'push' && github.ref == 'refs/heads/auto' && github.repository == 'rust-lang-ci/rust'"
steps:
- name: mark the job as a success
run: exit 0
shell: bash
name: bors build finished
runs-on: ubuntu-latest
auto-failure:
needs:
- auto
if: "!success() && github.event_name == 'push' && github.ref == 'refs/heads/auto' && github.repository == 'rust-lang-ci/rust'"
steps:
- name: mark the job as a failure
run: exit 1
shell: bash
name: bors build finished
runs-on: ubuntu-latest
Reference in New Issue View Git Blame Copy Permalink