Special thanks to @retep998 for the [excellent writeup](https://github.com/rust-lang/rfcs/issues/1061) of tasks to be done and @ricky26 for initially blazing the trail here!
# MSVC Support
This goal of this series of commits is to add MSVC support to the Rust compiler
and build system, allowing it more easily interoperate with Visual Studio
installations and native libraries compiled outside of MinGW.
The tl;dr; of this change is that there is a new target of the compiler,
`x86_64-pc-windows-msvc`, which will not interact with the MinGW toolchain at
all and will instead use `link.exe` to assemble output artifacts.
## Why try to use MSVC?
With today's Rust distribution, when you install a compiler on Windows you also
install `gcc.exe` and a number of supporting libraries by default (this can be
opted out of). This allows installations to remain independent of MinGW
installations, but it still generally requires native code to be linked with
MinGW instead of MSVC. Some more background can also be found in #1768 about the
incompatibilities between MinGW and MSVC.
Overall the current installation strategy is quite nice so long as you don't
interact with native code, but once you do the usage of a MinGW-based `gcc.exe`
starts to get quite painful.
Relying on a nonstandard Windows toolchain has also been a long-standing "code
smell" of Rust and has been slated for remedy for quite some time now. Using a
standard toolchain is a great motivational factor for improving the
interoperability of Rust code with the native system.
## What does it mean to use MSVC?
"Using MSVC" can be a bit of a nebulous concept, but this PR defines it as:
* The build system for Rust will build as much code as possible with the MSVC
compiler, `cl.exe`.
* The build system will use native MSVC tools for managing archives.
* The compiler will link all output with `link.exe` instead of `gcc.exe`.
None of these are currently implemented today, but all are required for the
compiler to fluently interoperate with MSVC.
## How does this all work?
At the highest level, this PR adds a new target triple to the Rust compiler:
x86_64-pc-windows-msvc
All logic for using MSVC or not is scoped within this triple and code can
conditionally build for MSVC or MinGW via:
#[cfg(target_env = "msvc")]
It is expected that auto builders will be set up for MSVC-based compiles in
addition to the existing MinGW-based compiles, and we will likely soon start
shipping MSVC nightlies where `x86_64-pc-windows-msvc` is the host target triple
of the compiler.
# Summary of changes
Here I'll explain at a high level what many of the changes made were targeted
at, but many more details can be found in the commits themselves. Many thanks to
@retep998 for the excellent writeup in rust-lang/rfcs#1061 and @rick26 for a lot
of the initial proof-of-concept work!
## Build system changes
As is probably expected, a large chunk of this PR is changes to Rust's build
system to build with MSVC. At a high level **it is an explicit non goal** to
enable building outside of a MinGW shell, instead all Makefile infrastructure we
have today is retrofitted with support to use MSVC instead of the standard MSVC
toolchain. Some of the high-level changes are:
* The configure script now detects when MSVC is being targeted and adds a number
of additional requirements about the build environment:
* The `--msvc-root` option must be specified or `cl.exe` must be in PATH to
discover where MSVC is installed. The compiler in use is also required to
target x86_64.
* Once the MSVC root is known, the INCLUDE/LIB environment variables are
scraped so they can be reexported by the build system.
* CMake is required to build LLVM with MSVC (and LLVM is also configured with
CMake instead of the normal configure script).
* jemalloc is currently unconditionally disabled for MSVC targets as jemalloc
isn't a hard requirement and I don't know how to build it with MSVC.
* Invocations of a C and/or C++ compiler are now abstracted behind macros to
appropriately call the underlying compiler with the correct format of
arguments, for example there is now a macro for "assemble an archive from
objects" instead of hard-coded invocations of `$(AR) crus liboutput.a ...`
* The output filenames for standard libraries such as morestack/compiler-rt are
now "more correct" on windows as they are shipped as `foo.lib` instead of
`libfoo.a`.
* Rust targets can now depend on native tools provided by LLVM, and as you'll
see in the commits the entire MSVC target depends on `llvm-ar.exe`.
* Support for custom arbitrary makefile dependencies of Rust targets has been
added. The MSVC target for `rustc_llvm` currently requires a custom `.DEF`
file to be passed to the linker to get further linkages to complete.
## Compiler changes
The modifications made to the compiler have so far largely been minor tweaks
here and there, mostly just adding a layer of abstraction over whether MSVC or a
GNU-like linker is being used. At a high-level these changes are:
* The section name for metadata storage in dynamic libraries is called `.rustc`
for MSVC-based platorms as section names cannot contain more than 8
characters.
* The implementation of `rustc_back::Archive` was refactored, but the
functionality has remained the same.
* Targets can now specify the default `ar` utility to use, and for MSVC this
defaults to `llvm-ar.exe`
* The building of the linker command in `rustc_trans:🔙:link` has been
abstracted behind a trait for the same code path to be used between GNU and
MSVC linkers.
## Standard library changes
Only a few small changes were required to the stadnard library itself, and only
for minor differences between the C runtime of msvcrt.dll and MinGW's libc.a
* Some function names for floating point functions have leading underscores, and
some are not present at all.
* Linkage to the `advapi32` library for crypto-related functions is now
explicit.
* Some small bits of C code here and there were fixed for compatibility with
MSVC's cl.exe compiler.
# Future Work
This commit is not yet a 100% complete port to using MSVC as there are still
some key components missing as well as some unimplemented optimizations. This PR
is already getting large enough that I wanted to draw the line here, but here's
a list of what is not implemented in this PR, on purpose:
## Unwinding
The revision of our LLVM submodule [does not seem to implement][llvm] does not
support lowering SEH exception handling on the Windows MSVC targets, so
unwinding support is not currently implemented for the standard library (it's
lowered to an abort).
[llvm]: https://github.com/rust-lang/llvm/blob/rust-llvm-2015-02-19/lib/CodeGen/Passes.cpp#L454-L461
It looks like, however, that upstream LLVM has quite a bit more support for SEH
unwinding and landing pads than the current revision we have, so adding support
will likely just involve updating LLVM and then adding some shims of our own
here and there.
## dllimport and dllexport
An interesting part of Windows which MSVC forces our hand on (and apparently
MinGW didn't) is the usage of `dllimport` and `dllexport` attributes in LLVM IR
as well as native dependencies (in C these correspond to
`__declspec(dllimport)`).
Whenever a dynamic library is built by MSVC it must have its public interface
specified by functions tagged with `dllexport` or otherwise they're not
available to be linked against. This poses a few problems for the compiler, some
of which are somewhat fundamental, but this commit alters the compiler to attach
the `dllexport` attribute to all LLVM functions that are reachable (e.g. they're
already tagged with external linkage). This is suboptimal for a few reasons:
* If an object file will never be included in a dynamic library, there's no need
to attach the dllexport attribute. Most object files in Rust are not destined
to become part of a dll as binaries are statically linked by default.
* If the compiler is emitting both an rlib and a dylib, the same source object
file is currently used but with MSVC this may be less feasible. The compiler
may be able to get around this, but it may involve some invasive changes to
deal with this.
The flipside of this situation is that whenever you link to a dll and you import
a function from it, the import should be tagged with `dllimport`. At this time,
however, the compiler does not emit `dllimport` for any declarations other than
constants (where it is required), which is again suboptimal for even more
reasons!
* Calling a function imported from another dll without using `dllimport` causes
the linker/compiler to have extra overhead (one `jmp` instruction on x86) when
calling the function.
* The same object file may be used in different circumstances, so a function may
be imported from a dll if the object is linked into a dll, but it may be
just linked against if linked into an rlib.
* The compiler has no knowledge about whether native functions should be tagged
dllimport or not.
For now the compiler takes the perf hit (I do not have any numbers to this
effect) by marking very little as `dllimport` and praying the linker will take
care of everything. Fixing this problem will likely require adding a few
attributes to Rust itself (feature gated at the start) and then strongly
recommending static linkage on Windows! This may also involve shipping a
statically linked compiler on Windows instead of a dynamically linked compiler,
but these sorts of changes are pretty invasive and aren't part of this PR.
## CI integration
Thankfully we don't need to set up a new snapshot bot for the changes made here as our snapshots are freestanding already, we should be able to use the same snapshot to bootstrap both MinGW and MSVC compilers (once a new snapshot is made from these changes).
I plan on setting up a new suite of auto bots which are testing MSVC configurations for now as well, for now they'll just be bootstrapping and not running tests, but once unwinding is implemented they'll start running all tests as well and we'll eventually start gating on them as well.
---
I'd love as many eyes on this as we've got as this was one of my first interactions with MSVC and Visual Studio, so there may be glaring holes that I'm missing here and there!
cc @retep998, @ricky26, @vadimcn, @klutzy
r? @brson
This commit was initially written to target either `ar` or `lib.exe` for MSVC,
but it ended up not needing `lib.exe` support after all. I would personally like
to refactor this to one day not invoke processes at all (but rather use the
`llvm-ar.cpp` file in LLVM as alibrary) so I chose to preserve this refactoring
to allow it to be easily done in the future.
I've been working with some archives generated by MSVC's `lib.exe` tool lately,
and it looks like the embedded name of the members in those archives sometimes
have slahes in the name (e.g. `foo/bar/baz.obj`). Currently the compiler chokes
on these paths as it assumes that each file in the archive is only the filename
(which is what unix does).
This commit interprets the name of each file in all archives as a path and then
only uses the `file_name` portion of the path to extract the file to a separate
location and then reassemble it back into a new archive later. Note that
duplicate filenames are already handled, so this won't introduce any conflicts.
When linking an archive statically to an rlib, the compiler will extract all
contents of the archive and add them all to the rlib being generated. The
current method of extraction is to run `ar x`, dumping all files into a
temporary directory. Object archives, however, are allowed to have multiple
entries with the same file name, so there is no method for them to extract their
contents into a directory in a lossless fashion.
This commit adds iterator support to the `ArchiveRO` structure which hooks into
LLVM's support for reading object archives. This iterator is then used to
inspect each object in turn and extract it to a unique location for later
assembly.
This commit:
* Introduces `std::convert`, providing an implementation of
RFC 529.
* Deprecates the `AsPath`, `AsOsStr`, and `IntoBytes` traits, all
in favor of the corresponding generic conversion traits.
Consequently, various IO APIs now take `AsRef<Path>` rather than
`AsPath`, and so on. Since the types provided by `std` implement both
traits, this should cause relatively little breakage.
* Deprecates many `from_foo` constructors in favor of `from`.
* Changes `PathBuf::new` to take no argument (creating an empty buffer,
as per convention). The previous behavior is now available as
`PathBuf::from`.
* De-stabilizes `IntoCow`. It's not clear whether we need this separate trait.
Closes#22751Closes#14433
[breaking-change]
This commit performs a stabilization pass over the `std::fs` module now that
it's had some time to bake. The change was largely just adding `#[stable]` tags,
but there are a few APIs that remain `#[unstable]`.
The following apis are now marked `#[stable]`:
* `std::fs` (the name)
* `File`
* `Metadata`
* `ReadDir`
* `DirEntry`
* `OpenOptions`
* `Permissions`
* `File::{open, create}`
* `File::{sync_all, sync_data}`
* `File::set_len`
* `File::metadata`
* Trait implementations for `File` and `&File`
* `OpenOptions::new`
* `OpenOptions::{read, write, append, truncate, create}`
* `OpenOptions::open` - this function was modified, however, to not attempt to
reject cross-platform openings of directories. This means that some platforms
will succeed in opening a directory and others will fail.
* `Metadata::{is_dir, is_file, len, permissions}`
* `Permissions::{readonly, set_readonly}`
* `Iterator for ReadDir`
* `DirEntry::path`
* `remove_file` - like with `OpenOptions::open`, the extra windows code to
remove a readonly file has been removed. This means that removing a readonly
file will succeed on some platforms but fail on others.
* `metadata`
* `rename`
* `copy`
* `hard_link`
* `soft_link`
* `read_link`
* `create_dir`
* `create_dir_all`
* `remove_dir`
* `remove_dir_all`
* `read_dir`
The following apis remain `#[unstable]`.
* `WalkDir` and `walk` - there are many methods by which a directory walk can be
constructed, and it's unclear whether the current semantics are the right
ones. For example symlinks are not handled super well currently. This is now
behind a new `fs_walk` feature.
* `File::path` - this is an extra abstraction which the standard library
provides on top of what the system offers and it's unclear whether we should
be doing so. This is now behind a new `file_path` feature.
* `Metadata::{accessed, modified}` - we do not currently have a good
abstraction for a moment in time which is what these APIs should likely be
returning, so these remain `#[unstable]` for now. These are now behind a new
`fs_time` feature
* `set_file_times` - like with `Metadata::accessed`, we do not currently have
the appropriate abstraction for the arguments here so this API remains
unstable behind the `fs_time` feature gate.
* `PathExt` - the precise set of methods on this trait may change over time and
some methods may be removed. This API remains unstable behind the `path_ext`
feature gate.
* `set_permissions` - we may wish to expose a more granular ability to set the
permissions on a file instead of just a blanket "set all permissions" method.
This function remains behind the `fs` feature.
The following apis are now `#[deprecated]`
* The `TempDir` type is now entirely deprecated and is [located on
crates.io][tempdir] as the `tempdir` crate with [its source][github] at
rust-lang/tempdir.
[tempdir]: https://crates.io/crates/tempdir
[github]: https://github.com/rust-lang/tempdir
The stability of some of these APIs has been questioned over the past few weeks
in using these APIs, and it is intentional that the majority of APIs here are
marked `#[stable]`. The `std::fs` module has a lot of room to grow and the
material is [being tracked in a RFC issue][rfc-issue].
[rfc-issue]: https://github.com/rust-lang/rfcs/issues/939
[breaking-change]
This commit deprecates the majority of std::old_io::fs in favor of std::fs and
its new functionality. Some functions remain non-deprecated but are now behind a
feature gate called `old_fs`. These functions will be deprecated once
suitable replacements have been implemented.
The compiler has been migrated to new `std::fs` and `std::path` APIs where
appropriate as part of this change.
This changes the type of some public constants/statics in libunicode.
Notably some `&'static &'static [(char, char)]` have changed
to `&'static [(char, char)]`. The regexp crate seems to be the
sole user of these, yet this is technically a [breaking-change]
This commit moves `std::env` away from the `std::old_io` error type as well as
the `std::old_path` module. Methods returning an error now return `io::Error`
and methods consuming or returning paths use `std::path` instead of
`std::old_path`. This commit does not yet mark these APIs as `#[stable]`.
This commit also migrates `std::old_io::TempDir` to `std::fs::TempDir` with
essentially the exact same API. This type was added to interoperate with the new
path API and has its own `tempdir` feature.
Finally, this commit reverts the deprecation of `std::os` APIs returning the old
path API types. This deprecation can come back once the entire `std::old_path`
module is deprecated.
[breaking-change]
This is an implementation of [RFC 578][rfc] which adds a new `std::env` module
to replace most of the functionality in the current `std::os` module. More
details can be found in the RFC itself, but as a summary the following methods
have all been deprecated:
[rfc]: https://github.com/rust-lang/rfcs/pull/578
* `os::args_as_bytes` => `env::args`
* `os::args` => `env::args`
* `os::consts` => `env::consts`
* `os::dll_filename` => no replacement, use `env::consts` directly
* `os::page_size` => `env::page_size`
* `os::make_absolute` => use `env::current_dir` + `join` instead
* `os::getcwd` => `env::current_dir`
* `os::change_dir` => `env::set_current_dir`
* `os::homedir` => `env::home_dir`
* `os::tmpdir` => `env::temp_dir`
* `os::join_paths` => `env::join_paths`
* `os::split_paths` => `env::split_paths`
* `os::self_exe_name` => `env::current_exe`
* `os::self_exe_path` => use `env::current_exe` + `pop`
* `os::set_exit_status` => `env::set_exit_status`
* `os::get_exit_status` => `env::get_exit_status`
* `os::env` => `env::vars`
* `os::env_as_bytes` => `env::vars`
* `os::getenv` => `env::var` or `env::var_string`
* `os::getenv_as_bytes` => `env::var`
* `os::setenv` => `env::set_var`
* `os::unsetenv` => `env::remove_var`
Many function signatures have also been tweaked for various purposes, but the
main changes were:
* `Vec`-returning APIs now all return iterators instead
* All APIs are now centered around `OsString` instead of `Vec<u8>` or `String`.
There is currently on convenience API, `env::var_string`, which can be used to
get the value of an environment variable as a unicode `String`.
All old APIs are `#[deprecated]` in-place and will remain for some time to allow
for migrations. The semantics of the APIs have been tweaked slightly with regard
to dealing with invalid unicode (panic instead of replacement).
The new `std::env` module is all contained within the `env` feature, so crates
must add the following to access the new APIs:
#![feature(env)]
[breaking-change]
This commit is an implementation of [RFC 565][rfc] which is a stabilization of
the `std::fmt` module and the implementations of various formatting traits.
Specifically, the following changes were performed:
[rfc]: https://github.com/rust-lang/rfcs/blob/master/text/0565-show-string-guidelines.md
* The `Show` trait is now deprecated, it was renamed to `Debug`
* The `String` trait is now deprecated, it was renamed to `Display`
* Many `Debug` and `Display` implementations were audited in accordance with the
RFC and audited implementations now have the `#[stable]` attribute
* Integers and floats no longer print a suffix
* Smart pointers no longer print details that they are a smart pointer
* Paths with `Debug` are now quoted and escape characters
* The `unwrap` methods on `Result` now require `Display` instead of `Debug`
* The `Error` trait no longer has a `detail` method and now requires that
`Display` must be implemented. With the loss of `String`, this has moved into
libcore.
* `impl<E: Error> FromError<E> for Box<Error>` now exists
* `derive(Show)` has been renamed to `derive(Debug)`. This is not currently
warned about due to warnings being emitted on stage1+
While backwards compatibility is attempted to be maintained with a blanket
implementation of `Display` for the old `String` trait (and the same for
`Show`/`Debug`) this is still a breaking change due to primitives no longer
implementing `String` as well as modifications such as `unwrap` and the `Error`
trait. Most code is fairly straightforward to update with a rename or tweaks of
method calls.
[breaking-change]
Closes#21436
fmt::Show is for debugging, and can and should be implemented for
all public types. This trait is used with `{:?}` syntax. There still
exists #[derive(Show)].
fmt::String is for types that faithfully be represented as a String.
Because of this, there is no way to derive fmt::String, all
implementations must be purposeful. It is used by the default format
syntax, `{}`.
This will break most instances of `{}`, since that now requires the type
to impl fmt::String. In most cases, replacing `{}` with `{:?}` is the
correct fix. Types that were being printed specifically for users should
receive a fmt::String implementation to fix this.
Part of #20013
[breaking-change]
os::getcwd() panics if the current directory is not available. According
to getcwd(3), there are three cases:
- EACCES: Permission denied.
- ENOENT: The current working directory has been removed.
- ERANGE: The buffer size is less than the actual absolute path.
This commit makes os::getcwd() return IoResult<Path>, not just Path,
preventing it from panicking.
As os::make_absolute() depends on os::getcwd(), it is also modified to
return IoResult<Path>.
Fixes#16946.
[breaking-change]
Removes all target-specific knowledge from rustc. Some targets have changed
during this, but none of these should be very visible outside of
cross-compilation. The changes make our targets more consistent.
iX86-unknown-linux-gnu is now only available as i686-unknown-linux-gnu. We
used to accept any value of X greater than 1. i686 was released in 1995, and
should encompass the bare minimum of what Rust supports on x86 CPUs.
The only two windows targets are now i686-pc-windows-gnu and
x86_64-pc-windows-gnu.
The iOS target has been renamed from arm-apple-ios to arm-apple-darwin.
A complete list of the targets we accept now:
arm-apple-darwin
arm-linux-androideabi
arm-unknown-linux-gnueabi
arm-unknown-linux-gnueabihf
i686-apple-darwin
i686-pc-windows-gnu
i686-unknown-freebsd
i686-unknown-linux-gnu
mips-unknown-linux-gnu
mipsel-unknown-linux-gnu
x86_64-apple-darwin
x86_64-unknown-freebsd
x86_64-unknown-linux-gnu
x86_64-pc-windows-gnu
Closes#16093
[breaking-change]
https://github.com/rust-lang/rfcs/pull/221
The current terminology of "task failure" often causes problems when
writing or speaking about code. You often want to talk about the
possibility of an operation that returns a Result "failing", but cannot
because of the ambiguity with task failure. Instead, you have to speak
of "the failing case" or "when the operation does not succeed" or other
circumlocutions.
Likewise, we use a "Failure" header in rustdoc to describe when
operations may fail the task, but it would often be helpful to separate
out a section describing the "Err-producing" case.
We have been steadily moving away from task failure and toward Result as
an error-handling mechanism, so we should optimize our terminology
accordingly: Result-producing functions should be easy to describe.
To update your code, rename any call to `fail!` to `panic!` instead.
Assuming you have not created your own macro named `panic!`, this
will work on UNIX based systems:
grep -lZR 'fail!' . | xargs -0 -l sed -i -e 's/fail!/panic!/g'
You can of course also do this by hand.
[breaking-change]
type they provide an implementation for.
This breaks code like:
mod foo {
struct Foo { ... }
}
impl foo::Foo {
...
}
Change this code to:
mod foo {
struct Foo { ... }
impl Foo {
...
}
}
Additionally, if you used the I/O path extension methods `stat`,
`lstat`, `exists`, `is_file`, or `is_dir`, note that these methods have
been moved to the the `std::io::fs::PathExtensions` trait. This breaks
code like:
fn is_it_there() -> bool {
Path::new("/foo/bar/baz").exists()
}
Change this code to:
use std::io::fs::PathExtensions;
fn is_it_there() -> bool {
Path::new("/foo/bar/baz").exists()
}
Closes#17059.
RFC #155.
[breaking-change]
For historical reasons, "Win32" has been used in Rust codebase to mean "Windows OS in general".
This is confusing, especially now, that Rust supports Win64 builds.
[breaking-change]
As discovered in #15460, a particular #[link(kind = "static", ...)] line is not
actually guaranteed to link the library at all. The reason for this is that if
the external library doesn't have any referenced symbols in the object generated
by rustc, the entire library is dropped by the linker.
For dynamic native libraries, this is solved by passing -lfoo for all downstream
compilations unconditionally. For static libraries in rlibs this is solved
because the entire archive is bundled in the rlib. The only situation in which
this was a problem was when a static native library was linked to a rust dynamic
library.
This commit brings the behavior of dylibs in line with rlibs by passing the
--whole-archive flag to the linker when linking native libraries. On OSX, this
uses the -force_load flag. This flag ensures that the entire archive is
considered candidate for being linked into the final dynamic library.
This is a breaking change because if any static library is included twice in the
same compilation unit then the linker will start emitting errors about duplicate
definitions now. The fix for this would involve only statically linking to a
library once.
Closes#15460
[breaking-change]
