rust/src/bootstrap/config.rs

687 lines
24 KiB
Rust
Raw Normal View History

//! Serialized configuration of a build.
//!
//! This module implements parsing `config.toml` configuration files to tweak
//! how the build runs.
use std::collections::{HashMap, HashSet};
use std::env;
use std::fs;
use std::path::{Path, PathBuf};
use std::process;
use std::cmp;
use build_helper::t;
2017-07-04 10:03:01 -06:00
use toml;
use serde::Deserialize;
2018-12-07 13:21:05 +01:00
use crate::cache::{INTERNER, Interned};
use crate::flags::Flags;
pub use crate::flags::Subcommand;
/// Global configuration for the entire build and/or bootstrap.
///
/// This structure is derived from a combination of both `config.toml` and
/// `config.mk`. As of the time of this writing it's unlikely that `config.toml`
/// is used all that much, so this is primarily filled out by `config.mk` which
/// is generated from `./configure`.
///
/// Note that this structure is not decoded directly into, but rather it is
/// filled out from the decoded forms of the structs below. For documentation
/// each field, see the corresponding fields in
/// `config.toml.example`.
#[derive(Default)]
pub struct Config {
pub ccache: Option<String>,
pub ninja: bool,
pub verbose: usize,
pub submodules: bool,
pub fast_submodules: bool,
pub compiler_docs: bool,
pub docs: bool,
pub locked_deps: bool,
pub vendor: bool,
pub target_config: HashMap<Interned<String>, Target>,
rustbuild: Compile rustc twice, not thrice This commit switches the rustbuild build system to compiling the compiler twice for a normal bootstrap rather than the historical three times. Rust is a bootstrapped language which means that a previous version of the compiler is used to build the next version of the compiler. Over time, however, we change many parts of compiler artifacts such as the metadata format, symbol names, etc. These changes make artifacts from one compiler incompatible from another compiler. Consequently if a compiler wants to be able to use some artifacts then it itself must have compiled the artifacts. Historically the rustc build system has achieved this by compiling the compiler three times: * An older compiler (stage0) is downloaded to kick off the chain. * This compiler now compiles a new compiler (stage1) * The stage1 compiler then compiles another compiler (stage2) * Finally, the stage2 compiler needs libraries to link against, so it compiles all the libraries again. This entire process amounts in compiling the compiler three times. Additionally, this process always guarantees that the Rust source tree can compile itself because the stage2 compiler (created by a freshly created compiler) would successfully compile itself again. This property, ensuring Rust can compile itself, is quite important! In general, though, this third compilation is not required for general purpose development on the compiler. The third compiler (stage2) can reuse the libraries that were created during the second compile. In other words, the second compilation can produce both a compiler and the libraries that compiler will use. These artifacts *must* be compatible due to the way plugins work today anyway, and they were created by the same source code so they *should* be compatible as well. So given all that, this commit switches the default build process to only compile the compiler three times, avoiding this third compilation by copying artifacts from the previous one. Along the way a new entry in the Travis matrix was also added to ensure that our full bootstrap can succeed. This entry does not run tests, though, as it should not be necessary. To restore the old behavior of a full bootstrap (three compiles) you can either pass: ./configure --enable-full-bootstrap or if you're using config.toml: [build] full-bootstrap = true Overall this will hopefully be an easy 33% win in build times of the compiler. If we do 33% less work we should be 33% faster! This in turn should affect cycle times and such on Travis and AppVeyor positively as well as making it easier to work on the compiler itself.
2016-12-25 15:20:33 -08:00
pub full_bootstrap: bool,
pub extended: bool,
pub tools: Option<HashSet<String>>,
pub sanitizers: bool,
pub profiler: bool,
pub ignore_git: bool,
pub exclude: Vec<PathBuf>,
2018-02-24 15:56:33 -07:00
pub rustc_error_format: Option<String>,
pub test_compare_mode: bool,
pub llvm_libunwind: bool,
pub skip_only_host_steps: bool,
pub on_fail: Option<String>,
pub stage: Option<u32>,
pub keep_stage: Vec<u32>,
pub src: PathBuf,
pub jobs: Option<u32>,
pub cmd: Subcommand,
pub incremental: bool,
pub dry_run: bool,
pub deny_warnings: bool,
2018-04-08 13:44:29 +02:00
pub backtrace_on_ice: bool,
// llvm codegen options
pub llvm_assertions: bool,
pub llvm_optimize: bool,
pub llvm_thin_lto: bool,
pub llvm_release_debuginfo: bool,
pub llvm_version_check: bool,
pub llvm_static_stdcpp: bool,
pub llvm_link_shared: bool,
pub llvm_clang_cl: Option<String>,
pub llvm_targets: Option<String>,
pub llvm_experimental_targets: Option<String>,
pub llvm_link_jobs: Option<u32>,
pub llvm_version_suffix: Option<String>,
pub llvm_use_linker: Option<String>,
pub llvm_allow_old_toolchain: Option<bool>,
rust: Import LLD for linking wasm objects This commit imports the LLD project from LLVM to serve as the default linker for the `wasm32-unknown-unknown` target. The `binaryen` submoule is consequently removed along with "binaryen linker" support in rustc. Moving to LLD brings with it a number of benefits for wasm code: * LLD is itself an actual linker, so there's no need to compile all wasm code with LTO any more. As a result builds should be *much* speedier as LTO is no longer forcibly enabled for all builds of the wasm target. * LLD is quickly becoming an "official solution" for linking wasm code together. This, I believe at least, is intended to be the main supported linker for native code and wasm moving forward. Picking up support early on should help ensure that we can help LLD identify bugs and otherwise prove that it works great for all our use cases! * Improvements to the wasm toolchain are currently primarily focused around LLVM and LLD (from what I can tell at least), so it's in general much better to be on this bandwagon for bugfixes and new features. * Historical "hacks" like `wasm-gc` will soon no longer be necessary, LLD will [natively implement][gc] `--gc-sections` (better than `wasm-gc`!) which means a postprocessor is no longer needed to show off Rust's "small wasm binary size". LLD is added in a pretty standard way to rustc right now. A new rustbuild target was defined for building LLD, and this is executed when a compiler's sysroot is being assembled. LLD is compiled against the LLVM that we've got in tree, which means we're currently on the `release_60` branch, but this may get upgraded in the near future! LLD is placed into rustc's sysroot in a `bin` directory. This is similar to where `gcc.exe` can be found on Windows. This directory is automatically added to `PATH` whenever rustc executes the linker, allowing us to define a `WasmLd` linker which implements the interface that `wasm-ld`, LLD's frontend, expects. Like Emscripten the LLD target is currently only enabled for Tier 1 platforms, notably OSX/Windows/Linux, and will need to be installed manually for compiling to wasm on other platforms. LLD is by default turned off in rustbuild, and requires a `config.toml` option to be enabled to turn it on. Finally the unstable `#![wasm_import_memory]` attribute was also removed as LLD has a native option for controlling this. [gc]: https://reviews.llvm.org/D42511
2017-08-26 18:30:12 -07:00
pub lld_enabled: bool,
pub lldb_enabled: bool,
2018-05-30 08:01:35 +02:00
pub llvm_tools_enabled: bool,
rust: Import LLD for linking wasm objects This commit imports the LLD project from LLVM to serve as the default linker for the `wasm32-unknown-unknown` target. The `binaryen` submoule is consequently removed along with "binaryen linker" support in rustc. Moving to LLD brings with it a number of benefits for wasm code: * LLD is itself an actual linker, so there's no need to compile all wasm code with LTO any more. As a result builds should be *much* speedier as LTO is no longer forcibly enabled for all builds of the wasm target. * LLD is quickly becoming an "official solution" for linking wasm code together. This, I believe at least, is intended to be the main supported linker for native code and wasm moving forward. Picking up support early on should help ensure that we can help LLD identify bugs and otherwise prove that it works great for all our use cases! * Improvements to the wasm toolchain are currently primarily focused around LLVM and LLD (from what I can tell at least), so it's in general much better to be on this bandwagon for bugfixes and new features. * Historical "hacks" like `wasm-gc` will soon no longer be necessary, LLD will [natively implement][gc] `--gc-sections` (better than `wasm-gc`!) which means a postprocessor is no longer needed to show off Rust's "small wasm binary size". LLD is added in a pretty standard way to rustc right now. A new rustbuild target was defined for building LLD, and this is executed when a compiler's sysroot is being assembled. LLD is compiled against the LLVM that we've got in tree, which means we're currently on the `release_60` branch, but this may get upgraded in the near future! LLD is placed into rustc's sysroot in a `bin` directory. This is similar to where `gcc.exe` can be found on Windows. This directory is automatically added to `PATH` whenever rustc executes the linker, allowing us to define a `WasmLd` linker which implements the interface that `wasm-ld`, LLD's frontend, expects. Like Emscripten the LLD target is currently only enabled for Tier 1 platforms, notably OSX/Windows/Linux, and will need to be installed manually for compiling to wasm on other platforms. LLD is by default turned off in rustbuild, and requires a `config.toml` option to be enabled to turn it on. Finally the unstable `#![wasm_import_memory]` attribute was also removed as LLD has a native option for controlling this. [gc]: https://reviews.llvm.org/D42511
2017-08-26 18:30:12 -07:00
pub llvm_cflags: Option<String>,
pub llvm_cxxflags: Option<String>,
pub llvm_ldflags: Option<String>,
pub llvm_use_libcxx: bool,
// rust codegen options
pub rust_optimize: bool,
pub rust_codegen_units: Option<u32>,
pub rust_codegen_units_std: Option<u32>,
pub rust_debug_assertions: bool,
pub rust_debuginfo_level_rustc: u32,
pub rust_debuginfo_level_std: u32,
pub rust_debuginfo_level_tools: u32,
pub rust_debuginfo_level_tests: u32,
pub rust_rpath: bool,
pub rustc_parallel: bool,
pub rustc_default_linker: Option<String>,
pub rust_optimize_tests: bool,
pub rust_dist_src: bool,
rustc: Split Emscripten to a separate codegen backend This commit introduces a separately compiled backend for Emscripten, avoiding compiling the `JSBackend` target in the main LLVM codegen backend. This builds on the foundation provided by #47671 to create a new codegen backend dedicated solely to Emscripten, removing the `JSBackend` of the main codegen backend in the process. A new field was added to each target for this commit which specifies the backend to use for translation, the default being `llvm` which is the main backend that we use. The Emscripten targets specify an `emscripten` backend instead of the main `llvm` one. There's a whole bunch of consequences of this change, but I'll try to enumerate them here: * A *second* LLVM submodule was added in this commit. The main LLVM submodule will soon start to drift from the Emscripten submodule, but currently they're both at the same revision. * Logic was added to rustbuild to *not* build the Emscripten backend by default. This is gated behind a `--enable-emscripten` flag to the configure script. By default users should neither check out the emscripten submodule nor compile it. * The `init_repo.sh` script was updated to fetch the Emscripten submodule from GitHub the same way we do the main LLVM submodule (a tarball fetch). * The Emscripten backend, turned off by default, is still turned on for a number of targets on CI. We'll only be shipping an Emscripten backend with Tier 1 platforms, though. All cross-compiled platforms will not be receiving an Emscripten backend yet. This commit means that when you download the `rustc` package in Rustup for Tier 1 platforms you'll be receiving two trans backends, one for Emscripten and one that's the general LLVM backend. If you never compile for Emscripten you'll never use the Emscripten backend, so we may update this one day to only download the Emscripten backend when you add the Emscripten target. For now though it's just an extra 10MB gzip'd. Closes #46819
2018-01-24 08:22:34 -08:00
pub rust_codegen_backends: Vec<Interned<String>>,
pub rust_codegen_backends_dir: String,
2018-06-12 21:21:29 +02:00
pub rust_verify_llvm_ir: bool,
pub rust_remap_debuginfo: bool,
pub build: Interned<String>,
pub hosts: Vec<Interned<String>>,
pub targets: Vec<Interned<String>>,
pub local_rebuild: bool,
pub jemalloc: bool,
// dist misc
pub dist_sign_folder: Option<PathBuf>,
pub dist_upload_addr: Option<String>,
pub dist_gpg_password_file: Option<PathBuf>,
// libstd features
pub backtrace: bool, // support for RUST_BACKTRACE
// misc
pub low_priority: bool,
pub channel: String,
2018-06-07 14:40:36 +02:00
pub verbose_tests: bool,
pub save_toolstates: Option<PathBuf>,
pub print_step_timings: bool,
2018-09-28 00:19:56 -05:00
pub missing_tools: bool,
// Fallback musl-root for all targets
pub musl_root: Option<PathBuf>,
pub prefix: Option<PathBuf>,
pub sysconfdir: Option<PathBuf>,
pub datadir: Option<PathBuf>,
pub docdir: Option<PathBuf>,
pub bindir: PathBuf,
pub libdir: Option<PathBuf>,
pub mandir: Option<PathBuf>,
pub codegen_tests: bool,
pub nodejs: Option<PathBuf>,
pub gdb: Option<PathBuf>,
pub python: Option<PathBuf>,
pub cargo_native_static: bool,
pub configure_args: Vec<String>,
// These are either the stage0 downloaded binaries or the locally installed ones.
pub initial_cargo: PathBuf,
pub initial_rustc: PathBuf,
pub out: PathBuf,
}
/// Per-target configuration stored in the global configuration structure.
#[derive(Default)]
pub struct Target {
/// Some(path to llvm-config) if using an external LLVM.
pub llvm_config: Option<PathBuf>,
/// Some(path to FileCheck) if one was specified.
pub llvm_filecheck: Option<PathBuf>,
pub cc: Option<PathBuf>,
pub cxx: Option<PathBuf>,
pub ar: Option<PathBuf>,
pub ranlib: Option<PathBuf>,
pub linker: Option<PathBuf>,
pub ndk: Option<PathBuf>,
pub crt_static: Option<bool>,
pub musl_root: Option<PathBuf>,
Add a new wasm32-unknown-wasi target This commit adds a new wasm32-based target distributed through rustup, supported in the standard library, and implemented in the compiler. The `wasm32-unknown-wasi` target is intended to be a WebAssembly target which matches the [WASI proposal recently announced.][LINK]. In summary the WASI target is an effort to define a standard set of syscalls for WebAssembly modules, allowing WebAssembly modules to not only be portable across architectures but also be portable across environments implementing this standard set of system calls. The wasi target in libstd is still somewhat bare bones. This PR does not fill out the filesystem, networking, threads, etc. Instead it only provides the most basic of integration with the wasi syscalls, enabling features like: * `Instant::now` and `SystemTime::now` work * `env::args` is hooked up * `env::vars` will look up environment variables * `println!` will print to standard out * `process::{exit, abort}` should be hooked up appropriately None of these APIs can work natively on the `wasm32-unknown-unknown` target, but with the assumption of the WASI set of syscalls we're able to provide implementations of these syscalls that engines can implement. Currently the primary engine implementing wasi is [wasmtime], but more will surely emerge! In terms of future development of libstd, I think this is something we'll probably want to discuss. The purpose of the WASI target is to provide a standardized set of syscalls, but it's *also* to provide a standard C sysroot for compiling C/C++ programs. This means it's intended that functions like `read` and `write` are implemented for this target with a relatively standard definition and implementation. It's unclear, therefore, how we want to expose file descriptors and how we'll want to implement system primitives. For example should `std::fs::File` have a libc-based file descriptor underneath it? The raw wasi file descriptor? We'll see! Currently these details are all intentionally hidden and things we can change over time. A `WasiFd` sample struct was added to the standard library as part of this commit, but it's not currently used. It shows how all the wasi syscalls could be ergonomically bound in Rust, and they offer a possible implementation of primitives like `std::fs::File` if we bind wasi file descriptors exactly. Apart from the standard library, there's also the matter of how this target is integrated with respect to its C standard library. The reference sysroot, for example, provides managment of standard unix file descriptors and also standard APIs like `open` (as opposed to the relative `openat` inspiration for the wasi ssycalls). Currently the standard library relies on the C sysroot symbols for operations such as environment management, process exit, and `read`/`write` of stdio fds. We want these operations in Rust to be interoperable with C if they're used in the same process. Put another way, if Rust and C are linked into the same WebAssembly binary they should work together, but that requires that the same C standard library is used. We also, however, want the `wasm32-unknown-wasi` target to be usable-by-default with the Rust compiler without requiring a separate toolchain to get downloaded and configured. With that in mind, there's two modes of operation for the `wasm32-unknown-wasi` target: 1. By default the C standard library is statically provided inside of `liblibc.rlib` distributed as part of the sysroot. This means that you can `rustc foo.wasm --target wasm32-unknown-unknown` and you're good to go, a fully workable wasi binary pops out. This is incompatible with linking in C code, however, which may be compiled against a different sysroot than the Rust code was previously compiled against. In this mode the default of `rust-lld` is used to link binaries. 2. For linking with C code, the `-C target-feature=-crt-static` flag needs to be passed. This takes inspiration from the musl target for this flag, but the idea is that you're no longer using the provided static C runtime, but rather one will be provided externally. This flag is intended to also get coupled with an external `clang` compiler configured with its own sysroot. Therefore you'll typically use this flag with `-C linker=/path/to/clang-script-wrapper`. Using this mode the Rust code will continue to reference standard C symbols, but the definition will be pulled in by the linker configured. Alright so that's all the current state of this PR. I suspect we'll definitely want to discuss this before landing of course! This PR is coupled with libc changes as well which I'll be posting shortly. [LINK]: [wasmtime]:
2019-02-13 10:02:22 -08:00
pub wasi_root: Option<PathBuf>,
pub qemu_rootfs: Option<PathBuf>,
pub no_std: bool,
}
/// Structure of the `config.toml` file that configuration is read from.
///
/// This structure uses `Decodable` to automatically decode a TOML configuration
/// file into this format, and then this is traversed and written into the above
/// `Config` structure.
2017-07-04 10:03:01 -06:00
#[derive(Deserialize, Default)]
#[serde(deny_unknown_fields, rename_all = "kebab-case")]
struct TomlConfig {
build: Option<Build>,
2016-12-19 15:49:57 -07:00
install: Option<Install>,
llvm: Option<Llvm>,
rust: Option<Rust>,
target: Option<HashMap<String, TomlTarget>>,
dist: Option<Dist>,
}
/// TOML representation of various global build decisions.
2017-07-04 10:03:01 -06:00
#[derive(Deserialize, Default, Clone)]
#[serde(deny_unknown_fields, rename_all = "kebab-case")]
struct Build {
build: Option<String>,
2017-07-04 10:03:01 -06:00
#[serde(default)]
host: Vec<String>,
2017-07-04 10:03:01 -06:00
#[serde(default)]
target: Vec<String>,
cargo: Option<String>,
rustc: Option<String>,
low_priority: Option<bool>,
compiler_docs: Option<bool>,
docs: Option<bool>,
submodules: Option<bool>,
fast_submodules: Option<bool>,
gdb: Option<String>,
locked_deps: Option<bool>,
vendor: Option<bool>,
nodejs: Option<String>,
python: Option<String>,
rustbuild: Compile rustc twice, not thrice This commit switches the rustbuild build system to compiling the compiler twice for a normal bootstrap rather than the historical three times. Rust is a bootstrapped language which means that a previous version of the compiler is used to build the next version of the compiler. Over time, however, we change many parts of compiler artifacts such as the metadata format, symbol names, etc. These changes make artifacts from one compiler incompatible from another compiler. Consequently if a compiler wants to be able to use some artifacts then it itself must have compiled the artifacts. Historically the rustc build system has achieved this by compiling the compiler three times: * An older compiler (stage0) is downloaded to kick off the chain. * This compiler now compiles a new compiler (stage1) * The stage1 compiler then compiles another compiler (stage2) * Finally, the stage2 compiler needs libraries to link against, so it compiles all the libraries again. This entire process amounts in compiling the compiler three times. Additionally, this process always guarantees that the Rust source tree can compile itself because the stage2 compiler (created by a freshly created compiler) would successfully compile itself again. This property, ensuring Rust can compile itself, is quite important! In general, though, this third compilation is not required for general purpose development on the compiler. The third compiler (stage2) can reuse the libraries that were created during the second compile. In other words, the second compilation can produce both a compiler and the libraries that compiler will use. These artifacts *must* be compatible due to the way plugins work today anyway, and they were created by the same source code so they *should* be compatible as well. So given all that, this commit switches the default build process to only compile the compiler three times, avoiding this third compilation by copying artifacts from the previous one. Along the way a new entry in the Travis matrix was also added to ensure that our full bootstrap can succeed. This entry does not run tests, though, as it should not be necessary. To restore the old behavior of a full bootstrap (three compiles) you can either pass: ./configure --enable-full-bootstrap or if you're using config.toml: [build] full-bootstrap = true Overall this will hopefully be an easy 33% win in build times of the compiler. If we do 33% less work we should be 33% faster! This in turn should affect cycle times and such on Travis and AppVeyor positively as well as making it easier to work on the compiler itself.
2016-12-25 15:20:33 -08:00
full_bootstrap: Option<bool>,
extended: Option<bool>,
tools: Option<HashSet<String>>,
verbose: Option<usize>,
sanitizers: Option<bool>,
profiler: Option<bool>,
cargo_native_static: Option<bool>,
configure_args: Option<Vec<String>>,
local_rebuild: Option<bool>,
print_step_timings: Option<bool>,
}
2016-12-19 15:49:57 -07:00
/// TOML representation of various global install decisions.
2017-07-04 10:03:01 -06:00
#[derive(Deserialize, Default, Clone)]
#[serde(deny_unknown_fields, rename_all = "kebab-case")]
2016-12-19 15:49:57 -07:00
struct Install {
prefix: Option<String>,
sysconfdir: Option<String>,
datadir: Option<String>,
docdir: Option<String>,
bindir: Option<String>,
libdir: Option<String>,
mandir: Option<String>,
// standard paths, currently unused
infodir: Option<String>,
localstatedir: Option<String>,
2016-12-19 15:49:57 -07:00
}
/// TOML representation of how the LLVM build is configured.
2017-07-04 10:03:01 -06:00
#[derive(Deserialize, Default)]
#[serde(deny_unknown_fields, rename_all = "kebab-case")]
struct Llvm {
optimize: Option<bool>,
thin_lto: Option<bool>,
release_debuginfo: Option<bool>,
assertions: Option<bool>,
ccache: Option<StringOrBool>,
version_check: Option<bool>,
static_libstdcpp: Option<bool>,
ninja: Option<bool>,
targets: Option<String>,
experimental_targets: Option<String>,
link_jobs: Option<u32>,
link_shared: Option<bool>,
version_suffix: Option<String>,
clang_cl: Option<String>,
cflags: Option<String>,
cxxflags: Option<String>,
ldflags: Option<String>,
use_libcxx: Option<bool>,
use_linker: Option<String>,
allow_old_toolchain: Option<bool>,
}
2017-07-04 10:03:01 -06:00
#[derive(Deserialize, Default, Clone)]
#[serde(deny_unknown_fields, rename_all = "kebab-case")]
struct Dist {
sign_folder: Option<String>,
gpg_password_file: Option<String>,
upload_addr: Option<String>,
src_tarball: Option<bool>,
missing_tools: Option<bool>,
}
2017-07-04 10:03:01 -06:00
#[derive(Deserialize)]
#[serde(untagged)]
enum StringOrBool {
String(String),
Bool(bool),
}
impl Default for StringOrBool {
fn default() -> StringOrBool {
StringOrBool::Bool(false)
}
}
/// TOML representation of how the Rust build is configured.
2017-07-04 10:03:01 -06:00
#[derive(Deserialize, Default)]
#[serde(deny_unknown_fields, rename_all = "kebab-case")]
struct Rust {
optimize: Option<bool>,
codegen_units: Option<u32>,
codegen_units_std: Option<u32>,
debug_assertions: Option<bool>,
debuginfo_level: Option<u32>,
debuginfo_level_rustc: Option<u32>,
debuginfo_level_std: Option<u32>,
debuginfo_level_tools: Option<u32>,
debuginfo_level_tests: Option<u32>,
parallel_compiler: Option<bool>,
backtrace: Option<bool>,
default_linker: Option<String>,
channel: Option<String>,
musl_root: Option<String>,
rpath: Option<bool>,
optimize_tests: Option<bool>,
codegen_tests: Option<bool>,
ignore_git: Option<bool>,
debug: Option<bool>,
dist_src: Option<bool>,
2018-06-07 14:40:36 +02:00
verbose_tests: Option<bool>,
incremental: Option<bool>,
save_toolstates: Option<String>,
rustc: Split Emscripten to a separate codegen backend This commit introduces a separately compiled backend for Emscripten, avoiding compiling the `JSBackend` target in the main LLVM codegen backend. This builds on the foundation provided by #47671 to create a new codegen backend dedicated solely to Emscripten, removing the `JSBackend` of the main codegen backend in the process. A new field was added to each target for this commit which specifies the backend to use for translation, the default being `llvm` which is the main backend that we use. The Emscripten targets specify an `emscripten` backend instead of the main `llvm` one. There's a whole bunch of consequences of this change, but I'll try to enumerate them here: * A *second* LLVM submodule was added in this commit. The main LLVM submodule will soon start to drift from the Emscripten submodule, but currently they're both at the same revision. * Logic was added to rustbuild to *not* build the Emscripten backend by default. This is gated behind a `--enable-emscripten` flag to the configure script. By default users should neither check out the emscripten submodule nor compile it. * The `init_repo.sh` script was updated to fetch the Emscripten submodule from GitHub the same way we do the main LLVM submodule (a tarball fetch). * The Emscripten backend, turned off by default, is still turned on for a number of targets on CI. We'll only be shipping an Emscripten backend with Tier 1 platforms, though. All cross-compiled platforms will not be receiving an Emscripten backend yet. This commit means that when you download the `rustc` package in Rustup for Tier 1 platforms you'll be receiving two trans backends, one for Emscripten and one that's the general LLVM backend. If you never compile for Emscripten you'll never use the Emscripten backend, so we may update this one day to only download the Emscripten backend when you add the Emscripten target. For now though it's just an extra 10MB gzip'd. Closes #46819
2018-01-24 08:22:34 -08:00
codegen_backends: Option<Vec<String>>,
codegen_backends_dir: Option<String>,
rust: Import LLD for linking wasm objects This commit imports the LLD project from LLVM to serve as the default linker for the `wasm32-unknown-unknown` target. The `binaryen` submoule is consequently removed along with "binaryen linker" support in rustc. Moving to LLD brings with it a number of benefits for wasm code: * LLD is itself an actual linker, so there's no need to compile all wasm code with LTO any more. As a result builds should be *much* speedier as LTO is no longer forcibly enabled for all builds of the wasm target. * LLD is quickly becoming an "official solution" for linking wasm code together. This, I believe at least, is intended to be the main supported linker for native code and wasm moving forward. Picking up support early on should help ensure that we can help LLD identify bugs and otherwise prove that it works great for all our use cases! * Improvements to the wasm toolchain are currently primarily focused around LLVM and LLD (from what I can tell at least), so it's in general much better to be on this bandwagon for bugfixes and new features. * Historical "hacks" like `wasm-gc` will soon no longer be necessary, LLD will [natively implement][gc] `--gc-sections` (better than `wasm-gc`!) which means a postprocessor is no longer needed to show off Rust's "small wasm binary size". LLD is added in a pretty standard way to rustc right now. A new rustbuild target was defined for building LLD, and this is executed when a compiler's sysroot is being assembled. LLD is compiled against the LLVM that we've got in tree, which means we're currently on the `release_60` branch, but this may get upgraded in the near future! LLD is placed into rustc's sysroot in a `bin` directory. This is similar to where `gcc.exe` can be found on Windows. This directory is automatically added to `PATH` whenever rustc executes the linker, allowing us to define a `WasmLd` linker which implements the interface that `wasm-ld`, LLD's frontend, expects. Like Emscripten the LLD target is currently only enabled for Tier 1 platforms, notably OSX/Windows/Linux, and will need to be installed manually for compiling to wasm on other platforms. LLD is by default turned off in rustbuild, and requires a `config.toml` option to be enabled to turn it on. Finally the unstable `#![wasm_import_memory]` attribute was also removed as LLD has a native option for controlling this. [gc]: https://reviews.llvm.org/D42511
2017-08-26 18:30:12 -07:00
lld: Option<bool>,
lldb: Option<bool>,
2018-04-30 10:15:48 +02:00
llvm_tools: Option<bool>,
deny_warnings: Option<bool>,
2018-04-08 13:44:29 +02:00
backtrace_on_ice: Option<bool>,
2018-06-12 21:21:29 +02:00
verify_llvm_ir: Option<bool>,
remap_debuginfo: Option<bool>,
jemalloc: Option<bool>,
test_compare_mode: Option<bool>,
llvm_libunwind: Option<bool>,
}
/// TOML representation of how each build target is configured.
2017-07-04 10:03:01 -06:00
#[derive(Deserialize, Default)]
#[serde(deny_unknown_fields, rename_all = "kebab-case")]
struct TomlTarget {
llvm_config: Option<String>,
llvm_filecheck: Option<String>,
cc: Option<String>,
cxx: Option<String>,
ar: Option<String>,
ranlib: Option<String>,
linker: Option<String>,
android_ndk: Option<String>,
crt_static: Option<bool>,
musl_root: Option<String>,
Add a new wasm32-unknown-wasi target This commit adds a new wasm32-based target distributed through rustup, supported in the standard library, and implemented in the compiler. The `wasm32-unknown-wasi` target is intended to be a WebAssembly target which matches the [WASI proposal recently announced.][LINK]. In summary the WASI target is an effort to define a standard set of syscalls for WebAssembly modules, allowing WebAssembly modules to not only be portable across architectures but also be portable across environments implementing this standard set of system calls. The wasi target in libstd is still somewhat bare bones. This PR does not fill out the filesystem, networking, threads, etc. Instead it only provides the most basic of integration with the wasi syscalls, enabling features like: * `Instant::now` and `SystemTime::now` work * `env::args` is hooked up * `env::vars` will look up environment variables * `println!` will print to standard out * `process::{exit, abort}` should be hooked up appropriately None of these APIs can work natively on the `wasm32-unknown-unknown` target, but with the assumption of the WASI set of syscalls we're able to provide implementations of these syscalls that engines can implement. Currently the primary engine implementing wasi is [wasmtime], but more will surely emerge! In terms of future development of libstd, I think this is something we'll probably want to discuss. The purpose of the WASI target is to provide a standardized set of syscalls, but it's *also* to provide a standard C sysroot for compiling C/C++ programs. This means it's intended that functions like `read` and `write` are implemented for this target with a relatively standard definition and implementation. It's unclear, therefore, how we want to expose file descriptors and how we'll want to implement system primitives. For example should `std::fs::File` have a libc-based file descriptor underneath it? The raw wasi file descriptor? We'll see! Currently these details are all intentionally hidden and things we can change over time. A `WasiFd` sample struct was added to the standard library as part of this commit, but it's not currently used. It shows how all the wasi syscalls could be ergonomically bound in Rust, and they offer a possible implementation of primitives like `std::fs::File` if we bind wasi file descriptors exactly. Apart from the standard library, there's also the matter of how this target is integrated with respect to its C standard library. The reference sysroot, for example, provides managment of standard unix file descriptors and also standard APIs like `open` (as opposed to the relative `openat` inspiration for the wasi ssycalls). Currently the standard library relies on the C sysroot symbols for operations such as environment management, process exit, and `read`/`write` of stdio fds. We want these operations in Rust to be interoperable with C if they're used in the same process. Put another way, if Rust and C are linked into the same WebAssembly binary they should work together, but that requires that the same C standard library is used. We also, however, want the `wasm32-unknown-wasi` target to be usable-by-default with the Rust compiler without requiring a separate toolchain to get downloaded and configured. With that in mind, there's two modes of operation for the `wasm32-unknown-wasi` target: 1. By default the C standard library is statically provided inside of `liblibc.rlib` distributed as part of the sysroot. This means that you can `rustc foo.wasm --target wasm32-unknown-unknown` and you're good to go, a fully workable wasi binary pops out. This is incompatible with linking in C code, however, which may be compiled against a different sysroot than the Rust code was previously compiled against. In this mode the default of `rust-lld` is used to link binaries. 2. For linking with C code, the `-C target-feature=-crt-static` flag needs to be passed. This takes inspiration from the musl target for this flag, but the idea is that you're no longer using the provided static C runtime, but rather one will be provided externally. This flag is intended to also get coupled with an external `clang` compiler configured with its own sysroot. Therefore you'll typically use this flag with `-C linker=/path/to/clang-script-wrapper`. Using this mode the Rust code will continue to reference standard C symbols, but the definition will be pulled in by the linker configured. Alright so that's all the current state of this PR. I suspect we'll definitely want to discuss this before landing of course! This PR is coupled with libc changes as well which I'll be posting shortly. [LINK]: [wasmtime]:
2019-02-13 10:02:22 -08:00
wasi_root: Option<String>,
qemu_rootfs: Option<String>,
}
impl Config {
fn path_from_python(var_key: &str) -> PathBuf {
match env::var_os(var_key) {
// Do not trust paths from Python and normalize them slightly (#49785).
Some(var_val) => Path::new(&var_val).components().collect(),
_ => panic!("expected '{}' to be set", var_key),
}
}
pub fn default_opts() -> Config {
let mut config = Config::default();
config.llvm_optimize = true;
config.llvm_version_check = true;
config.backtrace = true;
config.rust_optimize = true;
config.rust_optimize_tests = true;
config.submodules = true;
config.fast_submodules = true;
config.docs = true;
config.rust_rpath = true;
config.channel = "dev".to_string();
config.codegen_tests = true;
config.ignore_git = false;
config.rust_dist_src = true;
rustc: Split Emscripten to a separate codegen backend This commit introduces a separately compiled backend for Emscripten, avoiding compiling the `JSBackend` target in the main LLVM codegen backend. This builds on the foundation provided by #47671 to create a new codegen backend dedicated solely to Emscripten, removing the `JSBackend` of the main codegen backend in the process. A new field was added to each target for this commit which specifies the backend to use for translation, the default being `llvm` which is the main backend that we use. The Emscripten targets specify an `emscripten` backend instead of the main `llvm` one. There's a whole bunch of consequences of this change, but I'll try to enumerate them here: * A *second* LLVM submodule was added in this commit. The main LLVM submodule will soon start to drift from the Emscripten submodule, but currently they're both at the same revision. * Logic was added to rustbuild to *not* build the Emscripten backend by default. This is gated behind a `--enable-emscripten` flag to the configure script. By default users should neither check out the emscripten submodule nor compile it. * The `init_repo.sh` script was updated to fetch the Emscripten submodule from GitHub the same way we do the main LLVM submodule (a tarball fetch). * The Emscripten backend, turned off by default, is still turned on for a number of targets on CI. We'll only be shipping an Emscripten backend with Tier 1 platforms, though. All cross-compiled platforms will not be receiving an Emscripten backend yet. This commit means that when you download the `rustc` package in Rustup for Tier 1 platforms you'll be receiving two trans backends, one for Emscripten and one that's the general LLVM backend. If you never compile for Emscripten you'll never use the Emscripten backend, so we may update this one day to only download the Emscripten backend when you add the Emscripten target. For now though it's just an extra 10MB gzip'd. Closes #46819
2018-01-24 08:22:34 -08:00
config.rust_codegen_backends = vec![INTERNER.intern_str("llvm")];
config.rust_codegen_backends_dir = "codegen-backends".to_owned();
config.deny_warnings = true;
2018-09-28 00:19:56 -05:00
config.missing_tools = false;
// set by bootstrap.py
config.build = INTERNER.intern_str(&env::var("BUILD").expect("'BUILD' to be set"));
config.src = Config::path_from_python("SRC");
config.out = Config::path_from_python("BUILD_DIR");
config.initial_rustc = Config::path_from_python("RUSTC");
config.initial_cargo = Config::path_from_python("CARGO");
config
}
pub fn parse(args: &[String]) -> Config {
let flags = Flags::parse(&args);
let file = flags.config.clone();
let mut config = Config::default_opts();
config.exclude = flags.exclude;
2018-02-24 15:56:33 -07:00
config.rustc_error_format = flags.rustc_error_format;
config.on_fail = flags.on_fail;
config.stage = flags.stage;
config.jobs = flags.jobs.map(threads_from_config);
config.cmd = flags.cmd;
config.incremental = flags.incremental;
config.dry_run = flags.dry_run;
config.keep_stage = flags.keep_stage;
config.bindir = "bin".into(); // default
if let Some(value) = flags.deny_warnings {
config.deny_warnings = value;
}
if config.dry_run {
let dir = config.out.join("tmp-dry-run");
t!(fs::create_dir_all(&dir));
config.out = dir;
}
// If --target was specified but --host wasn't specified, don't run any host-only tests.
let has_hosts = !flags.host.is_empty();
let has_targets = !flags.target.is_empty();
config.skip_only_host_steps = !has_hosts && has_targets;
let toml = file.map(|file| {
let contents = t!(fs::read_to_string(&file));
2017-07-04 10:03:01 -06:00
match toml::from_str(&contents) {
Ok(table) => table,
Err(err) => {
println!("failed to parse TOML configuration '{}': {}",
file.display(), err);
process::exit(2);
}
}
}).unwrap_or_else(|| TomlConfig::default());
let build = toml.build.clone().unwrap_or_default();
// set by bootstrap.py
config.hosts.push(config.build.clone());
for host in build.host.iter() {
let host = INTERNER.intern_str(host);
if !config.hosts.contains(&host) {
config.hosts.push(host);
}
}
for target in config.hosts.iter().cloned()
.chain(build.target.iter().map(|s| INTERNER.intern_str(s)))
{
if !config.targets.contains(&target) {
config.targets.push(target);
}
}
config.hosts = if !flags.host.is_empty() {
flags.host
} else {
config.hosts
};
config.targets = if !flags.target.is_empty() {
flags.target
} else {
config.targets
};
config.nodejs = build.nodejs.map(PathBuf::from);
config.gdb = build.gdb.map(PathBuf::from);
config.python = build.python.map(PathBuf::from);
set(&mut config.low_priority, build.low_priority);
set(&mut config.compiler_docs, build.compiler_docs);
set(&mut config.docs, build.docs);
set(&mut config.submodules, build.submodules);
set(&mut config.fast_submodules, build.fast_submodules);
set(&mut config.locked_deps, build.locked_deps);
set(&mut config.vendor, build.vendor);
rustbuild: Compile rustc twice, not thrice This commit switches the rustbuild build system to compiling the compiler twice for a normal bootstrap rather than the historical three times. Rust is a bootstrapped language which means that a previous version of the compiler is used to build the next version of the compiler. Over time, however, we change many parts of compiler artifacts such as the metadata format, symbol names, etc. These changes make artifacts from one compiler incompatible from another compiler. Consequently if a compiler wants to be able to use some artifacts then it itself must have compiled the artifacts. Historically the rustc build system has achieved this by compiling the compiler three times: * An older compiler (stage0) is downloaded to kick off the chain. * This compiler now compiles a new compiler (stage1) * The stage1 compiler then compiles another compiler (stage2) * Finally, the stage2 compiler needs libraries to link against, so it compiles all the libraries again. This entire process amounts in compiling the compiler three times. Additionally, this process always guarantees that the Rust source tree can compile itself because the stage2 compiler (created by a freshly created compiler) would successfully compile itself again. This property, ensuring Rust can compile itself, is quite important! In general, though, this third compilation is not required for general purpose development on the compiler. The third compiler (stage2) can reuse the libraries that were created during the second compile. In other words, the second compilation can produce both a compiler and the libraries that compiler will use. These artifacts *must* be compatible due to the way plugins work today anyway, and they were created by the same source code so they *should* be compatible as well. So given all that, this commit switches the default build process to only compile the compiler three times, avoiding this third compilation by copying artifacts from the previous one. Along the way a new entry in the Travis matrix was also added to ensure that our full bootstrap can succeed. This entry does not run tests, though, as it should not be necessary. To restore the old behavior of a full bootstrap (three compiles) you can either pass: ./configure --enable-full-bootstrap or if you're using config.toml: [build] full-bootstrap = true Overall this will hopefully be an easy 33% win in build times of the compiler. If we do 33% less work we should be 33% faster! This in turn should affect cycle times and such on Travis and AppVeyor positively as well as making it easier to work on the compiler itself.
2016-12-25 15:20:33 -08:00
set(&mut config.full_bootstrap, build.full_bootstrap);
set(&mut config.extended, build.extended);
config.tools = build.tools;
set(&mut config.verbose, build.verbose);
set(&mut config.sanitizers, build.sanitizers);
set(&mut config.profiler, build.profiler);
set(&mut config.cargo_native_static, build.cargo_native_static);
set(&mut config.configure_args, build.configure_args);
set(&mut config.local_rebuild, build.local_rebuild);
set(&mut config.print_step_timings, build.print_step_timings);
config.verbose = cmp::max(config.verbose, flags.verbose);
2016-12-19 15:49:57 -07:00
if let Some(ref install) = toml.install {
config.prefix = install.prefix.clone().map(PathBuf::from);
config.sysconfdir = install.sysconfdir.clone().map(PathBuf::from);
config.datadir = install.datadir.clone().map(PathBuf::from);
config.docdir = install.docdir.clone().map(PathBuf::from);
set(&mut config.bindir, install.bindir.clone().map(PathBuf::from));
config.libdir = install.libdir.clone().map(PathBuf::from);
config.mandir = install.mandir.clone().map(PathBuf::from);
2016-12-19 15:49:57 -07:00
}
// Store off these values as options because if they're not provided
// we'll infer default values for them later
let mut llvm_assertions = None;
let mut debug = None;
let mut debug_assertions = None;
let mut debuginfo_level = None;
let mut debuginfo_level_rustc = None;
let mut debuginfo_level_std = None;
let mut debuginfo_level_tools = None;
let mut debuginfo_level_tests = None;
let mut optimize = None;
let mut ignore_git = None;
if let Some(ref llvm) = toml.llvm {
match llvm.ccache {
Some(StringOrBool::String(ref s)) => {
config.ccache = Some(s.to_string())
}
Some(StringOrBool::Bool(true)) => {
config.ccache = Some("ccache".to_string());
}
Some(StringOrBool::Bool(false)) | None => {}
}
set(&mut config.ninja, llvm.ninja);
llvm_assertions = llvm.assertions;
set(&mut config.llvm_optimize, llvm.optimize);
set(&mut config.llvm_thin_lto, llvm.thin_lto);
set(&mut config.llvm_release_debuginfo, llvm.release_debuginfo);
set(&mut config.llvm_version_check, llvm.version_check);
set(&mut config.llvm_static_stdcpp, llvm.static_libstdcpp);
set(&mut config.llvm_link_shared, llvm.link_shared);
config.llvm_targets = llvm.targets.clone();
config.llvm_experimental_targets = llvm.experimental_targets.clone();
config.llvm_link_jobs = llvm.link_jobs;
config.llvm_version_suffix = llvm.version_suffix.clone();
config.llvm_clang_cl = llvm.clang_cl.clone();
config.llvm_cflags = llvm.cflags.clone();
config.llvm_cxxflags = llvm.cxxflags.clone();
config.llvm_ldflags = llvm.ldflags.clone();
set(&mut config.llvm_use_libcxx, llvm.use_libcxx);
config.llvm_use_linker = llvm.use_linker.clone();
config.llvm_allow_old_toolchain = llvm.allow_old_toolchain.clone();
}
2016-12-19 17:42:07 -07:00
if let Some(ref rust) = toml.rust {
debug = rust.debug;
debug_assertions = rust.debug_assertions;
debuginfo_level = rust.debuginfo_level;
debuginfo_level_rustc = rust.debuginfo_level_rustc;
debuginfo_level_std = rust.debuginfo_level_std;
debuginfo_level_tools = rust.debuginfo_level_tools;
debuginfo_level_tests = rust.debuginfo_level_tests;
optimize = rust.optimize;
ignore_git = rust.ignore_git;
set(&mut config.rust_optimize_tests, rust.optimize_tests);
set(&mut config.codegen_tests, rust.codegen_tests);
set(&mut config.rust_rpath, rust.rpath);
set(&mut config.jemalloc, rust.jemalloc);
set(&mut config.test_compare_mode, rust.test_compare_mode);
set(&mut config.llvm_libunwind, rust.llvm_libunwind);
set(&mut config.backtrace, rust.backtrace);
set(&mut config.channel, rust.channel.clone());
set(&mut config.rust_dist_src, rust.dist_src);
2018-06-07 14:40:36 +02:00
set(&mut config.verbose_tests, rust.verbose_tests);
// in the case "false" is set explicitly, do not overwrite the command line args
if let Some(true) = rust.incremental {
config.incremental = true;
}
rust: Import LLD for linking wasm objects This commit imports the LLD project from LLVM to serve as the default linker for the `wasm32-unknown-unknown` target. The `binaryen` submoule is consequently removed along with "binaryen linker" support in rustc. Moving to LLD brings with it a number of benefits for wasm code: * LLD is itself an actual linker, so there's no need to compile all wasm code with LTO any more. As a result builds should be *much* speedier as LTO is no longer forcibly enabled for all builds of the wasm target. * LLD is quickly becoming an "official solution" for linking wasm code together. This, I believe at least, is intended to be the main supported linker for native code and wasm moving forward. Picking up support early on should help ensure that we can help LLD identify bugs and otherwise prove that it works great for all our use cases! * Improvements to the wasm toolchain are currently primarily focused around LLVM and LLD (from what I can tell at least), so it's in general much better to be on this bandwagon for bugfixes and new features. * Historical "hacks" like `wasm-gc` will soon no longer be necessary, LLD will [natively implement][gc] `--gc-sections` (better than `wasm-gc`!) which means a postprocessor is no longer needed to show off Rust's "small wasm binary size". LLD is added in a pretty standard way to rustc right now. A new rustbuild target was defined for building LLD, and this is executed when a compiler's sysroot is being assembled. LLD is compiled against the LLVM that we've got in tree, which means we're currently on the `release_60` branch, but this may get upgraded in the near future! LLD is placed into rustc's sysroot in a `bin` directory. This is similar to where `gcc.exe` can be found on Windows. This directory is automatically added to `PATH` whenever rustc executes the linker, allowing us to define a `WasmLd` linker which implements the interface that `wasm-ld`, LLD's frontend, expects. Like Emscripten the LLD target is currently only enabled for Tier 1 platforms, notably OSX/Windows/Linux, and will need to be installed manually for compiling to wasm on other platforms. LLD is by default turned off in rustbuild, and requires a `config.toml` option to be enabled to turn it on. Finally the unstable `#![wasm_import_memory]` attribute was also removed as LLD has a native option for controlling this. [gc]: https://reviews.llvm.org/D42511
2017-08-26 18:30:12 -07:00
set(&mut config.lld_enabled, rust.lld);
set(&mut config.lldb_enabled, rust.lldb);
2018-05-30 08:01:35 +02:00
set(&mut config.llvm_tools_enabled, rust.llvm_tools);
config.rustc_parallel = rust.parallel_compiler.unwrap_or(false);
config.rustc_default_linker = rust.default_linker.clone();
config.musl_root = rust.musl_root.clone().map(PathBuf::from);
config.save_toolstates = rust.save_toolstates.clone().map(PathBuf::from);
set(&mut config.deny_warnings, flags.deny_warnings.or(rust.deny_warnings));
2018-04-08 13:44:29 +02:00
set(&mut config.backtrace_on_ice, rust.backtrace_on_ice);
2018-06-12 21:21:29 +02:00
set(&mut config.rust_verify_llvm_ir, rust.verify_llvm_ir);
set(&mut config.rust_remap_debuginfo, rust.remap_debuginfo);
rustc: Split Emscripten to a separate codegen backend This commit introduces a separately compiled backend for Emscripten, avoiding compiling the `JSBackend` target in the main LLVM codegen backend. This builds on the foundation provided by #47671 to create a new codegen backend dedicated solely to Emscripten, removing the `JSBackend` of the main codegen backend in the process. A new field was added to each target for this commit which specifies the backend to use for translation, the default being `llvm` which is the main backend that we use. The Emscripten targets specify an `emscripten` backend instead of the main `llvm` one. There's a whole bunch of consequences of this change, but I'll try to enumerate them here: * A *second* LLVM submodule was added in this commit. The main LLVM submodule will soon start to drift from the Emscripten submodule, but currently they're both at the same revision. * Logic was added to rustbuild to *not* build the Emscripten backend by default. This is gated behind a `--enable-emscripten` flag to the configure script. By default users should neither check out the emscripten submodule nor compile it. * The `init_repo.sh` script was updated to fetch the Emscripten submodule from GitHub the same way we do the main LLVM submodule (a tarball fetch). * The Emscripten backend, turned off by default, is still turned on for a number of targets on CI. We'll only be shipping an Emscripten backend with Tier 1 platforms, though. All cross-compiled platforms will not be receiving an Emscripten backend yet. This commit means that when you download the `rustc` package in Rustup for Tier 1 platforms you'll be receiving two trans backends, one for Emscripten and one that's the general LLVM backend. If you never compile for Emscripten you'll never use the Emscripten backend, so we may update this one day to only download the Emscripten backend when you add the Emscripten target. For now though it's just an extra 10MB gzip'd. Closes #46819
2018-01-24 08:22:34 -08:00
if let Some(ref backends) = rust.codegen_backends {
config.rust_codegen_backends = backends.iter()
.map(|s| INTERNER.intern_str(s))
.collect();
}
set(&mut config.rust_codegen_backends_dir, rust.codegen_backends_dir.clone());
config.rust_codegen_units = rust.codegen_units.map(threads_from_config);
config.rust_codegen_units_std = rust.codegen_units_std.map(threads_from_config);
}
if let Some(ref t) = toml.target {
for (triple, cfg) in t {
let mut target = Target::default();
if let Some(ref s) = cfg.llvm_config {
target.llvm_config = Some(config.src.join(s));
}
if let Some(ref s) = cfg.llvm_filecheck {
target.llvm_filecheck = Some(config.src.join(s));
}
if let Some(ref s) = cfg.android_ndk {
target.ndk = Some(config.src.join(s));
}
target.cc = cfg.cc.clone().map(PathBuf::from);
target.cxx = cfg.cxx.clone().map(PathBuf::from);
target.ar = cfg.ar.clone().map(PathBuf::from);
target.ranlib = cfg.ranlib.clone().map(PathBuf::from);
target.linker = cfg.linker.clone().map(PathBuf::from);
target.crt_static = cfg.crt_static.clone();
target.musl_root = cfg.musl_root.clone().map(PathBuf::from);
Add a new wasm32-unknown-wasi target This commit adds a new wasm32-based target distributed through rustup, supported in the standard library, and implemented in the compiler. The `wasm32-unknown-wasi` target is intended to be a WebAssembly target which matches the [WASI proposal recently announced.][LINK]. In summary the WASI target is an effort to define a standard set of syscalls for WebAssembly modules, allowing WebAssembly modules to not only be portable across architectures but also be portable across environments implementing this standard set of system calls. The wasi target in libstd is still somewhat bare bones. This PR does not fill out the filesystem, networking, threads, etc. Instead it only provides the most basic of integration with the wasi syscalls, enabling features like: * `Instant::now` and `SystemTime::now` work * `env::args` is hooked up * `env::vars` will look up environment variables * `println!` will print to standard out * `process::{exit, abort}` should be hooked up appropriately None of these APIs can work natively on the `wasm32-unknown-unknown` target, but with the assumption of the WASI set of syscalls we're able to provide implementations of these syscalls that engines can implement. Currently the primary engine implementing wasi is [wasmtime], but more will surely emerge! In terms of future development of libstd, I think this is something we'll probably want to discuss. The purpose of the WASI target is to provide a standardized set of syscalls, but it's *also* to provide a standard C sysroot for compiling C/C++ programs. This means it's intended that functions like `read` and `write` are implemented for this target with a relatively standard definition and implementation. It's unclear, therefore, how we want to expose file descriptors and how we'll want to implement system primitives. For example should `std::fs::File` have a libc-based file descriptor underneath it? The raw wasi file descriptor? We'll see! Currently these details are all intentionally hidden and things we can change over time. A `WasiFd` sample struct was added to the standard library as part of this commit, but it's not currently used. It shows how all the wasi syscalls could be ergonomically bound in Rust, and they offer a possible implementation of primitives like `std::fs::File` if we bind wasi file descriptors exactly. Apart from the standard library, there's also the matter of how this target is integrated with respect to its C standard library. The reference sysroot, for example, provides managment of standard unix file descriptors and also standard APIs like `open` (as opposed to the relative `openat` inspiration for the wasi ssycalls). Currently the standard library relies on the C sysroot symbols for operations such as environment management, process exit, and `read`/`write` of stdio fds. We want these operations in Rust to be interoperable with C if they're used in the same process. Put another way, if Rust and C are linked into the same WebAssembly binary they should work together, but that requires that the same C standard library is used. We also, however, want the `wasm32-unknown-wasi` target to be usable-by-default with the Rust compiler without requiring a separate toolchain to get downloaded and configured. With that in mind, there's two modes of operation for the `wasm32-unknown-wasi` target: 1. By default the C standard library is statically provided inside of `liblibc.rlib` distributed as part of the sysroot. This means that you can `rustc foo.wasm --target wasm32-unknown-unknown` and you're good to go, a fully workable wasi binary pops out. This is incompatible with linking in C code, however, which may be compiled against a different sysroot than the Rust code was previously compiled against. In this mode the default of `rust-lld` is used to link binaries. 2. For linking with C code, the `-C target-feature=-crt-static` flag needs to be passed. This takes inspiration from the musl target for this flag, but the idea is that you're no longer using the provided static C runtime, but rather one will be provided externally. This flag is intended to also get coupled with an external `clang` compiler configured with its own sysroot. Therefore you'll typically use this flag with `-C linker=/path/to/clang-script-wrapper`. Using this mode the Rust code will continue to reference standard C symbols, but the definition will be pulled in by the linker configured. Alright so that's all the current state of this PR. I suspect we'll definitely want to discuss this before landing of course! This PR is coupled with libc changes as well which I'll be posting shortly. [LINK]: [wasmtime]:
2019-02-13 10:02:22 -08:00
target.wasi_root = cfg.wasi_root.clone().map(PathBuf::from);
target.qemu_rootfs = cfg.qemu_rootfs.clone().map(PathBuf::from);
config.target_config.insert(INTERNER.intern_string(triple.clone()), target);
}
}
if let Some(ref t) = toml.dist {
config.dist_sign_folder = t.sign_folder.clone().map(PathBuf::from);
config.dist_gpg_password_file = t.gpg_password_file.clone().map(PathBuf::from);
config.dist_upload_addr = t.upload_addr.clone();
set(&mut config.rust_dist_src, t.src_tarball);
set(&mut config.missing_tools, t.missing_tools);
}
// Now that we've reached the end of our configuration, infer the
// default values for all options that we haven't otherwise stored yet.
set(&mut config.initial_rustc, build.rustc.map(PathBuf::from));
2018-04-07 15:10:36 +01:00
set(&mut config.initial_cargo, build.cargo.map(PathBuf::from));
let default = false;
config.llvm_assertions = llvm_assertions.unwrap_or(default);
let default = true;
config.rust_optimize = optimize.unwrap_or(default);
let default = debug == Some(true);
config.rust_debug_assertions = debug_assertions.unwrap_or(default);
let with_defaults = |debuginfo_level_specific: Option<u32>| {
debuginfo_level_specific
.or(debuginfo_level)
.unwrap_or(if debug == Some(true) { 2 } else { 0 })
};
config.rust_debuginfo_level_rustc = with_defaults(debuginfo_level_rustc);
config.rust_debuginfo_level_std = with_defaults(debuginfo_level_std);
config.rust_debuginfo_level_tools = with_defaults(debuginfo_level_tools);
config.rust_debuginfo_level_tests = debuginfo_level_tests.unwrap_or(0);
let default = config.channel == "dev";
config.ignore_git = ignore_git.unwrap_or(default);
config
}
/// Try to find the relative path of `libdir`.
pub fn libdir_relative(&self) -> Option<&Path> {
let libdir = self.libdir.as_ref()?;
if libdir.is_relative() {
Some(libdir)
} else {
// Try to make it relative to the prefix.
libdir.strip_prefix(self.prefix.as_ref()?).ok()
}
}
pub fn verbose(&self) -> bool {
self.verbose > 0
}
pub fn very_verbose(&self) -> bool {
self.verbose > 1
}
2019-03-02 11:43:17 +01:00
pub fn llvm_enabled(&self) -> bool {
self.rust_codegen_backends.contains(&INTERNER.intern_str("llvm"))
|| self.rust_codegen_backends.contains(&INTERNER.intern_str("emscripten"))
}
}
fn set<T>(field: &mut T, val: Option<T>) {
if let Some(v) = val {
*field = v;
}
}
fn threads_from_config(v: u32) -> u32 {
match v {
0 => num_cpus::get() as u32,
n => n,
}
}