mikros/rust
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Auto merge of #79540 - jyn514:no-xpy, r=Mark-Simulacrum

Browse Source 
Allow building rustdoc without first building rustc (MVP)

 ## Motivation

The compile times for rustc are extremely long and a major issue for
recruiting new contributors to rustdoc. People interested in joining
often give up after running into issues with submodules or python
versions. stage1 rustdoc fundamentally doesn't care about bootstrapping
or stages, it just needs `rustc_private` available.

## Summary of Changes

- Add an opt-in `[rust] download_rustc` option
- Determine the version of the compiler to download using `log --author=bors`
- Do no work for any component other than `Rustdoc` for any stage. Instead, copy the CI artifacts from the downloaded sysroot stage0/ to stage0-sysroot/ or stage1/ in `Sysroot`. This is done with an `ENABLE_DOWNLOAD_STAGE1` constant which is off by default.
- Don't download different versions of rustfmt or cargo - those should still use the beta version (rustfmt especially).

The vast majority of work is done in bootstrap.py, which downloads the artifacts and extracts them to stage0/ in place of the beta compiler. Rustbuild just takes care of copying the artifacts to stage1 if necessary.

## Future work

- I turned off verification for the commit tarballs because the .sha256 URLs gave a 404. This seems not ideal, it would be nice to start signing them.
- This will break if you rebase an old enough branch (I think commits are kept at most 160 days?). This doesn't need to be supported, but it would be nice to give a reasonable error. https://github.com/rust-lang/rust/pull/79540#issuecomment-751481291
- Right now, every time you rebase, stage0 tools (bootstrap, tidy, ...) will have to be recompiled. Additionally running `x.py setup tools` will compile rustbuild twice. Instead, this should download a separate beta compiler for stage0 and only use CI artifacts for stage1 onward. https://github.com/rust-lang/rust/pull/79540#issuecomment-757047321
- Add `x.py setup tools` to enable this conveniently (it doesn't make sense to use this for compiler developers). cb5d8c8522
- Compile a new version of tracing so that rustdoc still gets debug logging (since CI artifacts always disable `debug` and `trace` logging). https://github.com/rust-lang/rust/pull/79540#issuecomment-742756411, 6a5d512420
- Right now only rustdoc is ever rebuilt. This is not ideal and should probably at least compile compiler tools (rustfmt, clippy, miri). https://github.com/rust-lang/rust/pull/79540#issuecomment-775634693
- Using `git log --author=bors` sometimes breaks. This should use `git merge-base` instead. https://github.com/rust-lang/rust/pull/79540#discussion_r572572280
- It would be nice to support cross-compiling the standard library. Right now this gives an assertion failure I think.

Some of this work has already been done in (the history for) 673476c785.
This commit is contained in:
bors 2021-02-09 15:28:28 +00:00
commit 185de5f41a
7 changed files with 122 additions and 18 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
config.toml.example
View File

@ -358,6 +358,12 @@ changelog-seen = 2
#
#debug = false
# Whether to download the stage 1 and 2 compilers from CI.
# This is mostly useful for tools; if you have changes to `compiler/` they will be ignored.
#
# FIXME: currently, this also uses the downloaded compiler for stage0, but that causes unnecessary rebuilds.
#download-rustc = false
# Number of codegen units to use for each compiler invocation. A value of 0
# means "the number of cores on this machine", and 1+ is passed through to the
# compiler.

74
src/bootstrap/bootstrap.py
View File

@ -378,6 +378,7 @@ class RustBuild(object):
self.verbose = False
self.git_version = None
self.nix_deps_dir = None
self.rustc_commit = None
def download_stage0(self):
"""Fetch the build system for Rust, written in Rust
@ -394,20 +395,27 @@ class RustBuild(object):
if self.rustc().startswith(self.bin_root()) and \
(not os.path.exists(self.rustc()) or
self.program_out_of_date(self.rustc_stamp(), self.date)):
self.program_out_of_date(self.rustc_stamp(), self.date + str(self.rustc_commit))):
if os.path.exists(self.bin_root()):
shutil.rmtree(self.bin_root())
download_rustc = self.rustc_commit is not None
tarball_suffix = '.tar.xz' if support_xz() else '.tar.gz'
filename = "rust-std-{}-{}{}".format(
rustc_channel, self.build, tarball_suffix)
pattern = "rust-std-{}".format(self.build)
self._download_stage0_helper(filename, pattern, tarball_suffix)
self._download_component_helper(filename, pattern, tarball_suffix, download_rustc)
filename = "rustc-{}-{}{}".format(rustc_channel, self.build,
tarball_suffix)
self._download_stage0_helper(filename, "rustc", tarball_suffix)
self._download_component_helper(filename, "rustc", tarball_suffix, download_rustc)
filename = "cargo-{}-{}{}".format(rustc_channel, self.build,
tarball_suffix)
self._download_stage0_helper(filename, "cargo", tarball_suffix)
self._download_component_helper(filename, "cargo", tarball_suffix)
if self.rustc_commit is not None:
filename = "rustc-dev-{}-{}{}".format(rustc_channel, self.build, tarball_suffix)
self._download_component_helper(
filename, "rustc-dev", tarball_suffix, download_rustc
)
self.fix_bin_or_dylib("{}/bin/rustc".format(self.bin_root()))
self.fix_bin_or_dylib("{}/bin/rustdoc".format(self.bin_root()))
self.fix_bin_or_dylib("{}/bin/cargo".format(self.bin_root()))
@ -416,7 +424,7 @@ class RustBuild(object):
if lib.endswith(".so"):
self.fix_bin_or_dylib(os.path.join(lib_dir, lib), rpath_libz=True)
with output(self.rustc_stamp()) as rust_stamp:
rust_stamp.write(self.date)
rust_stamp.write(self.date + str(self.rustc_commit))
if self.rustfmt() and self.rustfmt().startswith(self.bin_root()) and (
not os.path.exists(self.rustfmt())
@ -426,7 +434,9 @@ class RustBuild(object):
tarball_suffix = '.tar.xz' if support_xz() else '.tar.gz'
[channel, date] = rustfmt_channel.split('-', 1)
filename = "rustfmt-{}-{}{}".format(channel, self.build, tarball_suffix)
self._download_stage0_helper(filename, "rustfmt-preview", tarball_suffix, date)
self._download_component_helper(
filename, "rustfmt-preview", tarball_suffix, key=date
)
self.fix_bin_or_dylib("{}/bin/rustfmt".format(self.bin_root()))
self.fix_bin_or_dylib("{}/bin/cargo-fmt".format(self.bin_root()))
with output(self.rustfmt_stamp()) as rustfmt_stamp:
@ -482,18 +492,27 @@ class RustBuild(object):
return opt == "true" \
or (opt == "if-available" and self.build in supported_platforms)
def _download_stage0_helper(self, filename, pattern, tarball_suffix, date=None):
if date is None:
date = self.date
def _download_component_helper(
self, filename, pattern, tarball_suffix, download_rustc=False, key=None
):
if key is None:
if download_rustc:
key = self.rustc_commit
else:
key = self.date
cache_dst = os.path.join(self.build_dir, "cache")
rustc_cache = os.path.join(cache_dst, date)
rustc_cache = os.path.join(cache_dst, key)
if not os.path.exists(rustc_cache):
os.makedirs(rustc_cache)
url = "{}/dist/{}".format(self._download_url, date)
if download_rustc:
url = "https://ci-artifacts.rust-lang.org/rustc-builds/{}".format(self.rustc_commit)
else:
url = "{}/dist/{}".format(self._download_url, key)
tarball = os.path.join(rustc_cache, filename)
if not os.path.exists(tarball):
get("{}/{}".format(url, filename), tarball, verbose=self.verbose)
do_verify = not download_rustc
get("{}/{}".format(url, filename), tarball, verbose=self.verbose, do_verify=do_verify)
unpack(tarball, tarball_suffix, self.bin_root(), match=pattern, verbose=self.verbose)
def _download_ci_llvm(self, llvm_sha, llvm_assertions):
@ -613,6 +632,30 @@ class RustBuild(object):
print("warning: failed to call patchelf:", reason)
return
# Return the stage1 compiler to download, if any.
def maybe_download_rustc(self):
# If `download-rustc` is not set, default to rebuilding.
if self.get_toml("download-rustc", section="rust") != "true":
return None
# Handle running from a directory other than the top level
rev_parse = ["git", "rev-parse", "--show-toplevel"]
top_level = subprocess.check_output(rev_parse, universal_newlines=True).strip()
compiler = "{}/compiler/".format(top_level)
# Look for a version to compare to based on the current commit.
# Ideally this would just use `merge-base`, but on beta and stable branches that wouldn't
# come up with any commits, so hack it and use `author=bors` instead.
merge_base = ["git", "log", "--author=bors", "--pretty=%H", "-n1", "--", compiler]
commit = subprocess.check_output(merge_base, universal_newlines=True).strip()
# Warn if there were changes to the compiler since the ancestor commit.
status = subprocess.call(["git", "diff-index", "--quiet", commit, "--", compiler])
if status != 0:
print("warning: `download-rustc` is enabled, but there are changes to compiler/")
return commit
def rustc_stamp(self):
"""Return the path for .rustc-stamp
@ -1090,6 +1133,13 @@ def bootstrap(help_triggered):
build.update_submodules()
# Fetch/build the bootstrap
build.rustc_commit = build.maybe_download_rustc()
if build.rustc_commit is not None:
if build.verbose:
commit = build.rustc_commit
print("using downloaded stage1 artifacts from CI (commit {})".format(commit))
# FIXME: support downloading artifacts from the beta channel
build.rustc_channel = "nightly"
build.download_stage0()
sys.stdout.flush()
build.ensure_vendored()

26
src/bootstrap/builder.rs
View File

@ -57,6 +57,14 @@ pub trait Step: 'static + Clone + Debug + PartialEq + Eq + Hash {
/// `true` here can still be overwritten by `should_run` calling `default_condition`.
const DEFAULT: bool = false;
/// Whether this step should be run even when `download-rustc` is set.
///
/// Most steps are not important when the compiler is downloaded, since they will be included in
/// the pre-compiled sysroot. Steps can set this to `true` to be built anyway.
///
/// When in doubt, set this to `false`.
const ENABLE_DOWNLOAD_RUSTC: bool = false;
/// If true, then this rule should be skipped if --target was specified, but --host was not
const ONLY_HOSTS: bool = false;
@ -99,6 +107,7 @@ pub fn build_triple(&self) -> TargetSelection {
struct StepDescription {
default: bool,
enable_download_rustc: bool,
only_hosts: bool,
should_run: fn(ShouldRun<'_>) -> ShouldRun<'_>,
make_run: fn(RunConfig<'_>),
@ -153,6 +162,7 @@ impl StepDescription {
fn from<S: Step>() -> StepDescription {
StepDescription {
default: S::DEFAULT,
enable_download_rustc: S::ENABLE_DOWNLOAD_RUSTC,
only_hosts: S::ONLY_HOSTS,
should_run: S::should_run,
make_run: S::make_run,
@ -169,6 +179,14 @@ fn maybe_run(&self, builder: &Builder<'_>, pathset: &PathSet) {
"{:?} not skipped for {:?} -- not in {:?}",
pathset, self.name, builder.config.exclude
);
} else if builder.config.download_rustc && !self.enable_download_rustc {
if !builder.config.dry_run {
eprintln!(
"Not running {} because its artifacts have been downloaded from CI (`download-rustc` is set)",
self.name
);
}
return;
}
// Determine the targets participating in this rule.
@ -629,8 +647,12 @@ fn run(self, builder: &Builder<'_>) -> Interned<PathBuf> {
.join("rustlib")
.join(self.target.triple)
.join("lib");
let _ = fs::remove_dir_all(&sysroot);
t!(fs::create_dir_all(&sysroot));
// Avoid deleting the rustlib/ directory we just copied
// (in `impl Step for Sysroot`).
if !builder.config.download_rustc {
let _ = fs::remove_dir_all(&sysroot);
t!(fs::create_dir_all(&sysroot));
}
INTERNER.intern_path(sysroot)
}
}

4
src/bootstrap/check.rs
View File

@ -62,6 +62,7 @@ fn cargo_subcommand(kind: Kind) -> &'static str {
impl Step for Std {
type Output = ();
const DEFAULT: bool = true;
const ENABLE_DOWNLOAD_RUSTC: bool = true;
fn should_run(run: ShouldRun<'_>) -> ShouldRun<'_> {
run.all_krates("test")
@ -155,6 +156,7 @@ impl Step for Rustc {
type Output = ();
const ONLY_HOSTS: bool = true;
const DEFAULT: bool = true;
const ENABLE_DOWNLOAD_RUSTC: bool = true;
fn should_run(run: ShouldRun<'_>) -> ShouldRun<'_> {
run.all_krates("rustc-main")
@ -233,6 +235,7 @@ impl Step for CodegenBackend {
type Output = ();
const ONLY_HOSTS: bool = true;
const DEFAULT: bool = true;
const ENABLE_DOWNLOAD_RUSTC: bool = true;
fn should_run(run: ShouldRun<'_>) -> ShouldRun<'_> {
run.paths(&["compiler/rustc_codegen_cranelift", "rustc_codegen_cranelift"])
@ -290,6 +293,7 @@ impl Step for $name {
type Output = ();
const ONLY_HOSTS: bool = true;
const DEFAULT: bool = true;
const ENABLE_DOWNLOAD_RUSTC: bool = true;
fn should_run(run: ShouldRun<'_>) -> ShouldRun<'_> {
run.path($path)

26
src/bootstrap/compile.rs
View File

@ -41,7 +41,10 @@ impl Step for Std {
const DEFAULT: bool = true;
fn should_run(run: ShouldRun<'_>) -> ShouldRun<'_> {
run.all_krates("test")
// When downloading stage1, the standard library has already been copied to the sysroot, so
// there's no need to rebuild it.
let download_rustc = run.builder.config.download_rustc;
run.all_krates("test").default_condition(!download_rustc)
}
fn make_run(run: RunConfig<'_>) {
@ -904,6 +907,18 @@ fn run(self, builder: &Builder<'_>) -> Interned<PathBuf> {
let _ = fs::remove_dir_all(&sysroot);
t!(fs::create_dir_all(&sysroot));
// If we're downloading a compiler from CI, we can use the same compiler for all stages other than 0.
if builder.config.download_rustc {
assert_eq!(
builder.config.build, compiler.host,
"Cross-compiling is not yet supported with `download-rustc`",
);
// Copy the compiler into the correct sysroot.
let stage0_dir = builder.config.out.join(&*builder.config.build.triple).join("stage0");
builder.cp_r(&stage0_dir, &sysroot);
return INTERNER.intern_path(sysroot);
}
// Symlink the source root into the same location inside the sysroot,
// where `rust-src` component would go (`$sysroot/lib/rustlib/src/rust`),
// so that any tools relying on `rust-src` also work for local builds,
@ -975,13 +990,16 @@ fn run(self, builder: &Builder<'_>) -> Compiler {
// produce some other architecture compiler we need to start from
// `build` to get there.
//
// FIXME: Perhaps we should download those libraries?
// It would make builds faster...
//
// FIXME: It may be faster if we build just a stage 1 compiler and then
// use that to bootstrap this compiler forward.
let build_compiler = builder.compiler(target_compiler.stage - 1, builder.config.build);
// If we're downloading a compiler from CI, we can use the same compiler for all stages other than 0.
if builder.config.download_rustc {
builder.ensure(Sysroot { compiler: target_compiler });
return target_compiler;
}
// Build the libraries for this compiler to link to (i.e., the libraries
// it uses at runtime). NOTE: Crates the target compiler compiles don't
// link to these. (FIXME: Is that correct? It seems to be correct most

3
src/bootstrap/config.rs
View File

@ -80,6 +80,7 @@ pub struct Config {
pub cmd: Subcommand,
pub incremental: bool,
pub dry_run: bool,
pub download_rustc: bool,
pub deny_warnings: bool,
pub backtrace_on_ice: bool,
@ -503,6 +504,7 @@ struct Rust {
new_symbol_mangling: Option<bool>,
profile_generate: Option<String>,
profile_use: Option<String>,
download_rustc: Option<bool>,
}
/// TOML representation of how each build target is configured.
@ -885,6 +887,7 @@ pub fn parse(args: &[String]) -> Config {
config.rust_codegen_units_std = rust.codegen_units_std.map(threads_from_config);
config.rust_profile_use = flags.rust_profile_use.or(rust.profile_use);
config.rust_profile_generate = flags.rust_profile_generate.or(rust.profile_generate);
config.download_rustc = rust.download_rustc.unwrap_or(false);
} else {
config.rust_profile_use = flags.rust_profile_use;
config.rust_profile_generate = flags.rust_profile_generate;

1
src/bootstrap/tool.rs
View File

@ -477,6 +477,7 @@ pub struct Rustdoc {
impl Step for Rustdoc {
type Output = PathBuf;
const DEFAULT: bool = true;
const ENABLE_DOWNLOAD_RUSTC: bool = true;
const ONLY_HOSTS: bool = true;
fn should_run(run: ShouldRun<'_>) -> ShouldRun<'_> {