mikros/rust
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Auto merge of #121182 - majaha:mingw_ci_new, r=Mark-Simulacrum

Browse Source 
Improvements to building and CI for mingw/msys

I was getting error messages when trying to follow the build instructions the mingw build for Rust, and managed to track the issue down to an incomparability of Rust's bootstrap program with MSYS2's version of git. Essentially, the problem is that MSYS2's git works in emulated unix-y paths, but bootstrap expects a Windows path. I found a workaround for this by using relative paths instead of absolute paths.

Along with that fix, this PR also updates the build instructions for MinGW to be compatible with modern versions of MSYS2, and some changes to CI to make sure that MSYS2's version of git is tested. In particular, I'm suggesting using the [MSYS2 github action](https://github.com/marketplace/actions/setup-msys2) specially made for this purpose, which is much less hacky than the old approach and gives us more control of what packages are installed. I also cleaned up as many alternate versions of key tools as I could find from PATH, to avoid accidental usage, and cleaned up some abuses of the `CUSTOM_MINGW` environment variable.

This fixes https://github.com/rust-lang/rust/issues/105696 and fixes https://github.com/rust-lang/rust/issues/117567
This commit is contained in:
bors 2024-02-25 21:10:15 +00:00
commit 0ecbd06057
9 changed files with 132 additions and 32 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

33
.github/workflows/ci.yml vendored
View File

@ -65,9 +65,20 @@ jobs:
- name: x86_64-gnu-tools
os: ubuntu-20.04-16core-64gb
env: {}
defaults:
run:
shell: "${{ contains(matrix.os, 'windows') && 'msys2 {0}' || 'bash' }}"
timeout-minutes: 600
runs-on: "${{ matrix.os }}"
steps:
- if: "contains(matrix.os, 'windows')"
uses: msys2/setup-msys2@v2.22.0
with:
msystem: "${{ contains(matrix.name, 'i686') && 'mingw32' || 'mingw64' }}"
update: false
release: true
path-type: inherit
install: "make dos2unix diffutils\n"
- name: disable git crlf conversion
run: git config --global core.autocrlf false
- name: checkout the source code
@ -461,9 +472,20 @@ jobs:
RUST_CONFIGURE_ARGS: "--build=x86_64-pc-windows-msvc --enable-extended --enable-profiler"
SCRIPT: python x.py dist bootstrap --include-default-paths
os: windows-2019-8core-32gb
defaults:
run:
shell: "${{ contains(matrix.os, 'windows') && 'msys2 {0}' || 'bash' }}"
timeout-minutes: 600
runs-on: "${{ matrix.os }}"
steps:
- if: "contains(matrix.os, 'windows')"
uses: msys2/setup-msys2@v2.22.0
with:
msystem: "${{ contains(matrix.name, 'i686') && 'mingw32' || 'mingw64' }}"
update: false
release: true
path-type: inherit
install: "make dos2unix diffutils\n"
- name: disable git crlf conversion
run: git config --global core.autocrlf false
- name: checkout the source code
@ -589,9 +611,20 @@ jobs:
env:
CODEGEN_BACKENDS: "llvm,cranelift"
os: ubuntu-20.04-16core-64gb
defaults:
run:
shell: "${{ contains(matrix.os, 'windows') && 'msys2 {0}' || 'bash' }}"
timeout-minutes: 600
runs-on: "${{ matrix.os }}"
steps:
- if: "contains(matrix.os, 'windows')"
uses: msys2/setup-msys2@v2.22.0
with:
msystem: "${{ contains(matrix.name, 'i686') && 'mingw32' || 'mingw64' }}"
update: false
release: true
path-type: inherit
install: "make dos2unix diffutils\n"
- name: disable git crlf conversion
run: git config --global core.autocrlf false
- name: checkout the source code

24
INSTALL.md
View File

@ -145,10 +145,9 @@ toolchain.
1. Download the latest [MSYS2 installer][msys2] and go through the installer.
2. Run `mingw32_shell.bat` or `mingw64_shell.bat` from the MSYS2 installation
directory (e.g. `C:\msys64`), depending on whether you want 32-bit or 64-bit
Rust. (As of the latest version of MSYS2 you have to run `msys2_shell.cmd
-mingw32` or `msys2_shell.cmd -mingw64` from the command line instead.)
2. Start a MINGW64 or MINGW32 shell (depending on whether you want 32-bit
or 64-bit Rust) either from your start menu, or by running `mingw64.exe`
or `mingw32.exe` from your MSYS2 installation directory (e.g. `C:\msys64`).
3. From this terminal, install the required tools:
@ -157,8 +156,7 @@ toolchain.
pacman -Sy pacman-mirrors
# Install build tools needed for Rust. If you're building a 32-bit compiler,
# then replace "x86_64" below with "i686". If you've already got Git, Python,
# or CMake installed and in PATH you can remove them from this list.
# then replace "x86_64" below with "i686".
# Note that it is important that you do **not** use the 'python2', 'cmake',
# and 'ninja' packages from the 'msys2' subsystem.
# The build has historically been known to fail with these packages.
@ -175,9 +173,21 @@ toolchain.
4. Navigate to Rust's source code (or clone it), then build it:
```sh
python x.py setup user && python x.py build && python x.py install
python x.py setup dist && python x.py build && python x.py install
```
If you want to use the native versions of Git, Python, or CMake you can remove
them from the above pacman command and install them from another source. Make
sure that they're in your Windows PATH, and edit the relevant `mingw[32|64].ini`
file in your MSYS2 installation directory by uncommenting the line
`MSYS2_PATH_TYPE=inherit` to include them in your MSYS2 PATH.
Using Windows native Python can be helpful if you get errors when building LLVM.
You may also want to use Git for Windows, as it is often *much* faster. Turning
off real-time protection in the Windows Virus & Threat protections settings can
also help with long run times (although note that it will automatically turn
itself back on after some time).
### MSVC
MSVC builds of Rust additionally require an installation of Visual Studio 2017

29
src/bootstrap/src/core/config/config.rs
View File

@ -1235,12 +1235,16 @@ pub(crate) fn parse_inner(args: &[String], get_toml: impl Fn(&Path) -> TomlConfi
// Infer the rest of the configuration.
// Infer the source directory. This is non-trivial because we want to support a downloaded bootstrap binary,
// running on a completely machine from where it was compiled.
// running on a completely different machine from where it was compiled.
let mut cmd = Command::new("git");
// NOTE: we cannot support running from outside the repository because the only path we have available
// is set at compile time, which can be wrong if bootstrap was downloaded from source.
// NOTE: we cannot support running from outside the repository because the only other path we have available
// is set at compile time, which can be wrong if bootstrap was downloaded rather than compiled locally.
// We still support running outside the repository if we find we aren't in a git directory.
cmd.arg("rev-parse").arg("--show-toplevel");
// NOTE: We get a relative path from git to work around an issue on MSYS/mingw. If we used an absolute path,
// and end up using MSYS's git rather than git-for-windows, we would get a unix-y MSYS path. But as bootstrap
// has already been (kinda-cross-)compiled to Windows land, we require a normal Windows path.
cmd.arg("rev-parse").arg("--show-cdup");
// Discard stderr because we expect this to fail when building from a tarball.
let output = cmd
.stderr(std::process::Stdio::null())
@ -1248,13 +1252,18 @@ pub(crate) fn parse_inner(args: &[String], get_toml: impl Fn(&Path) -> TomlConfi
.ok()
.and_then(|output| if output.status.success() { Some(output) } else { None });
if let Some(output) = output {
let git_root = String::from_utf8(output.stdout).unwrap();
// We need to canonicalize this path to make sure it uses backslashes instead of forward slashes.
let git_root = PathBuf::from(git_root.trim()).canonicalize().unwrap();
let git_root_relative = String::from_utf8(output.stdout).unwrap();
// We need to canonicalize this path to make sure it uses backslashes instead of forward slashes,
// and to resolve any relative components.
let git_root = env::current_dir()
.unwrap()
.join(PathBuf::from(git_root_relative.trim()))
.canonicalize()
.unwrap();
let s = git_root.to_str().unwrap();
// Bootstrap is quite bad at handling /? in front of paths
let src = match s.strip_prefix("\\\\?\\") {
let git_root = match s.strip_prefix("\\\\?\\") {
Some(p) => PathBuf::from(p),
None => git_root,
};
@ -1264,8 +1273,8 @@ pub(crate) fn parse_inner(args: &[String], get_toml: impl Fn(&Path) -> TomlConfi
//
// NOTE: this implies that downloadable bootstrap isn't supported when the build directory is outside
// the source directory. We could fix that by setting a variable from all three of python, ./x, and x.ps1.
if src.join("src").join("stage0.json").exists() {
config.src = src;
if git_root.join("src").join("stage0.json").exists() {
config.src = git_root;
}
} else {
// We're building from a tarball, not git sources.

21
src/ci/github-actions/ci.yml
View File

@ -111,10 +111,31 @@ x--expand-yaml-anchors--remove:
if: success() && !env.SKIP_JOB
- &base-ci-job
defaults:
run:
shell: ${{ contains(matrix.os, 'windows') && 'msys2 {0}' || 'bash' }}
timeout-minutes: 600
runs-on: "${{ matrix.os }}"
env: *shared-ci-variables
steps:
- if: contains(matrix.os, 'windows')
uses: msys2/setup-msys2@v2.22.0
with:
# i686 jobs use mingw32. x86_64 and cross-compile jobs use mingw64.
msystem: ${{ contains(matrix.name, 'i686') && 'mingw32' || 'mingw64' }}
# don't try to download updates for already installed packages
update: false
# don't try to use the msys that comes built-in to the github runner,
# so we can control what is installed (i.e. not python)
release: true
# Inherit the full path from the Windows environment, with MSYS2's */bin/
# dirs placed in front. This lets us run Windows-native Python etc.
path-type: inherit
install: >
make
dos2unix
diffutils
- name: disable git crlf conversion
run: git config --global core.autocrlf false

2
src/ci/run.sh
View File

@ -76,7 +76,7 @@ RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --set dist.compression-profile=balance
# the LLVM build, as not to run out of memory.
# This is an attempt to fix the spurious build error tracked by
# https://github.com/rust-lang/rust/issues/108227.
if isWindows && [[ ${CUSTOM_MINGW-0} -eq 1 ]]; then
if isKnownToBeMingwBuild; then
RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --set llvm.link-jobs=1"
fi

3
src/ci/scripts/install-clang.sh
View File

@ -37,8 +37,7 @@ if isMacOS; then
# Configure `AR` specifically so rustbuild doesn't try to infer it as
# `clang-ar` by accident.
ciCommandSetEnv AR "ar"
elif isWindows && [[ ${CUSTOM_MINGW-0} -ne 1 ]]; then
elif isWindows && ! isKnownToBeMingwBuild; then
# If we're compiling for MSVC then we, like most other distribution builders,
# switch to clang as the compiler. This'll allow us eventually to enable LTO
# amongst LLVM and rustc. Note that we only do this on MSVC as I don't think

10
src/ci/scripts/install-mingw.sh
View File

@ -38,11 +38,11 @@ if isWindows; then
;;
esac
if [[ "${CUSTOM_MINGW-0}" -ne 1 ]]; then
pacman -S --noconfirm --needed mingw-w64-$arch-toolchain mingw-w64-$arch-cmake \
mingw-w64-$arch-gcc \
mingw-w64-$arch-python # the python package is actually for python3
ciCommandAddPath "$(ciCheckoutPath)/msys2/mingw${bits}/bin"
if [[ "${CUSTOM_MINGW:-0}" == 0 ]]; then
pacboy -S --noconfirm toolchain:p
# According to the comment in the Windows part of install-clang.sh, in the future we might
# want to do this instead:
# pacboy -S --noconfirm clang:p ...
else
mingw_dir="mingw${bits}"

38
src/ci/scripts/install-msys2.sh
View File

@ -1,17 +1,12 @@
#!/bin/bash
# Download and install MSYS2, needed primarily for the test suite (run-make) but
# also used by the MinGW toolchain for assembling things.
# Clean up and prepare the MSYS2 installation. MSYS2 is needed primarily for
# the test suite (run-make), but is also used by the MinGW toolchain for assembling things.
set -euo pipefail
IFS=$'\n\t'
source "$(cd "$(dirname "$0")" && pwd)/../shared.sh"
if isWindows; then
msys2Path="c:/msys64"
mkdir -p "${msys2Path}/home/${USERNAME}"
ciCommandAddPath "${msys2Path}/usr/bin"
# Detect the native Python version installed on the agent. On GitHub
# Actions, the C:\hostedtoolcache\windows\Python directory contains a
# subdirectory for each installed Python version.
@ -29,4 +24,33 @@ if isWindows; then
fi
ciCommandAddPath "C:\\hostedtoolcache\\windows\\Python\\${native_python_version}\\x64"
ciCommandAddPath "C:\\hostedtoolcache\\windows\\Python\\${native_python_version}\\x64\\Scripts"
# Install pacboy for easily installing packages
pacman -S --noconfirm pactoys
# Delete these pre-installed tools so we can't accidentally use them, because we are using the
# MSYS2 setup action versions instead.
# Delete pre-installed version of MSYS2
rm -r "/c/msys64/"
# Delete Strawberry Perl, which contains a version of mingw
rm -r "/c/Strawberry/"
# Delete these other copies of mingw, I don't even know where they come from.
rm -r "/c/mingw64/"
rm -r "/c/mingw32/"
if isKnownToBeMingwBuild; then
# Use the mingw version of CMake for mingw builds.
# However, the MSVC build needs native CMake, as it fails with the mingw one.
# Delete native CMake
rm -r "/c/Program Files/CMake/"
# Install mingw-w64-$arch-cmake
pacboy -S --noconfirm cmake:p
# We use Git-for-Windows for MSVC builds, and MSYS2 Git for mingw builds,
# so that both are tested.
# Delete Windows-Git
rm -r "/c/Program Files/Git/"
# Install MSYS2 git
pacman -S --noconfirm git
fi
fi

4
src/ci/shared.sh
View File

@ -52,6 +52,10 @@ function isLinux {
[[ "${OSTYPE}" = "linux-gnu" ]]
}
function isKnownToBeMingwBuild {
isGitHubActions && [[ "${CI_JOB_NAME}" == *mingw ]]
}
function isCiBranch {
if [[ $# -ne 1 ]]; then
echo "usage: $0 <branch-name>"