* Added doc comments explaining what all public functionality does.
* Added the ability to spawn a detached thread
* Added the ability for the procs to return a value in 'join'
Whenever the runtime is shut down, add a few hooks to clean up some of the
statically initialized data of the runtime. Note that this is an unsafe
operation because there's no guarantee on behalf of the runtime that there's no
other code running which is using the runtime.
This helps turn down the noise a bit in the valgrind output related to
statically initialized mutexes. It doesn't turn the noise down to 0 because
there are still statically initialized mutexes in dynamic_lib and
os::with_env_lock, but I believe that it would be easy enough to add exceptions
for those cases and I don't think that it's the runtime's job to go and clean up
that data.
This patchset fixes some parts broken on Win64.
This also adds `--disable-pthreads` flags to llvm on mingw-w64 archs (both 32-bit and 64-bit, not mingw) due to bad performance. See #8996 for discussion.
The reasons for doing this are:
* The model on which linked failure is based is inherently complex
* The implementation is also very complex, and there are few remaining who
fully understand the implementation
* There are existing race conditions in the core context switching function of
the scheduler, and possibly others.
* It's unclear whether this model of linked failure maps well to a 1:1 threading
model
Linked failure is often a desired aspect of tasks, but we would like to take a
much more conservative approach in re-implementing linked failure if at all.
Closes#8674Closes#8318Closes#8863
The reasons for doing this are:
* The model on which linked failure is based is inherently complex
* The implementation is also very complex, and there are few remaining who
fully understand the implementation
* There are existing race conditions in the core context switching function of
the scheduler, and possibly others.
* It's unclear whether this model of linked failure maps well to a 1:1 threading
model
Linked failure is often a desired aspect of tasks, but we would like to take a
much more conservative approach in re-implementing linked failure if at all.
Closes#8674Closes#8318Closes#8863
I cannot tell whether the original comment was unsure about the
arithmetic calculations, or if it was unsure about the assumptions
being made about the alignment of the current allocation pointer.
The arithmetic calculation looks fine to me, though. This technique
is documented e.g. in Henry Warren's "Hacker's Delight" (section 3-1).
(I am sure one can find it elsewhere too, its not an obscure
property.)
There are issues with reading stdin when it is actually attached to a pipe, but
I have run into no problems in writing to stdout/stderr when they are attached
to pipes.
Explicitly have the only C++ portion of the runtime be one file with exception
handling. All other runtime files must now live in C and be fully defined in C.
There are issues with reading stdin when it is actually attached to a pipe, but
I have run into no problems in writing to stdout/stderr when they are attached
to pipes.
This commit re-organizes the io::native module slightly in order to have a
working implementation of rtio::IoFactory which uses native implementations. The
goal is to seamlessly multiplex among libuv/native implementations wherever
necessary.
Right now most of the native I/O is unimplemented, but we have existing bindings
for file descriptors and processes which have been hooked up. What this means is
that you can now invoke println!() from libstd with no local task, no local
scheduler, and even without libuv.
There's still plenty of work to do on the native I/O factory, but this is the
first steps into making it an official portion of the standard library. I don't
expect anyone to reach into io::native directly, but rather only std::io
primitives will be used. Each std::io interface seamlessly falls back onto the
native I/O implementation if the local scheduler doesn't have a libuv one
(hurray trait ojects!)
I was benchmarking rust-http recently, and I saw that 50% of its time was spent
creating buffered readers/writers. Albeit rust-http wasn't using
std::rt::io::buffered, but the same idea applies here. It's much cheaper to
malloc a large region and not initialize it than to set it all to 0. Buffered
readers/writers never use uninitialized data, and their internal buffers are
encapsulated, so any usage of uninitialized slots are an implementation bug in
the readers/writers.
I increased this to 4MB when I implemented abort-on-stack-overflow for Rust
functions. Now that the fixed_stack_segment attribute is removed, no rust
function will ever reasonably request 2MB of stack (due to calling an FFI
function).
The default size of 2MB should be plenty for everyday use-cases, and tasks can
still request more stack via the spawning API.
These two attributes are no longer useful now that Rust has decided to leave
segmented stacks behind. It is assumed that the rust task's stack is always
large enough to make an FFI call (due to the stack being very large).
There's always the case of stack overflow, however, to consider. This does not
change the behavior of stack overflow in Rust. This is still normally triggered
by the __morestack function and aborts the whole process.
C stack overflow will continue to corrupt the stack, however (as it did before
this commit as well). The future improvement of a guard page at the end of every
rust stack is still unimplemented and is intended to be the mechanism through
which we attempt to detect C stack overflow.
Closes#8822Closes#10155
I was benchmarking rust-http recently, and I saw that 50% of its time was spent
creating buffered readers/writers. Albeit rust-http wasn't using
std::rt::io::buffered, but the same idea applies here. It's much cheaper to
malloc a large region and not initialize it than to set it all to 0. Buffered
readers/writers never use uninitialized data, and their internal buffers are
encapsulated, so any usage of uninitialized slots are an implementation bug in
the readers/writers.
It appears that uv's support for interacting with a stdio stream as a tty when
it's actually a pipe is pretty problematic. To get around this, promote a check
to see if the stream is a tty to the top of the tty constructor, and bail out
quickly if it's not identified as a tty.
Closes#10237
It turns out that the uv implementation would cause use-after-free if the idle
callback was used after the call to `close`, and additionally nothing would ever
really work that well if `start()` were called twice. To change this, the
`start` and `close` methods were removed in favor of specifying the callback at
creation, and allowing destruction to take care of closing the watcher.
This binds to the appropriate pthreads_* and Windows specific functions
and calls them from Rust. This allows for removal of the C++ support
code for threads.
Fixes#10162
Right now if you're running a program with its output piped to some location and
the program decides to go awry, when you kill the program via some signal none
of the program's last 4K of output will get printed to the screen. In theory the
solution to this would be to register a signal handler as part of the runtime
which then flushes the output stream.
I believe that the current behavior is far enough from what's expected that we
shouldn't be providing this sort of "super buffering" by default when stdout
isn't attached to a tty.
This isn't quite as fancy as the struct in #9913, but I'm not sure we should be exposing crate names/hashes of the types. That being said, it'd be pretty easy to extend this (the deterministic hashing regardless of what crate you're in was the hard part).
Right now if you're running a program with its output piped to some location and
the program decides to go awry, when you kill the program via some signal none
of the program's last 4K of output will get printed to the screen. In theory the
solution to this would be to register a signal handler as part of the runtime
which then flushes the output stream.
I believe that the current behavior is far enough from what's expected that we
shouldn't be providing this sort of "super buffering" by default when stdout
isn't attached to a tty.
This renames the `file` module to `fs` because that more accurately describes
its current purpose (manipulating the filesystem, not just files).
Additionally, this adds an UnstableFileStat structure as a nested structure of
FileStat to signify that the fields should not be depended on. The structure is
currently flagged with #[unstable], but it's unlikely that it has much meaning.
Closes#10241
This adds bindings to the remaining functions provided by libuv, all of which
are useful operations on files which need to get exposed somehow.
Some highlights:
* Dropped `FileReader` and `FileWriter` and `FileStream` for one `File` type
* Moved all file-related methods to be static methods under `File`
* All directory related methods are still top-level functions
* Created `io::FilePermission` types (backed by u32) that are what you'd expect
* Created `io::FileType` and refactored `FileStat` to use FileType and
FilePermission
* Removed the expanding matrix of `FileMode` operations. The mode of reading a
file will not have the O_CREAT flag, but a write mode will always have the
O_CREAT flag.
Closes#10130Closes#10131Closes#10121
This commit moves all thread-blocking I/O functions from the std::os module.
Their replacements can be found in either std::rt::io::file or in a hidden
"old_os" module inside of native::file. I didn't want to outright delete these
functions because they have a lot of special casing learned over time for each
OS/platform, and I imagine that these will someday get integrated into a
blocking implementation of IoFactory. For now, they're moved to a private module
to prevent bitrot and still have tests to ensure that they work.
I've also expanded the extensions to a few more methods defined on Path, most of
which were previously defined in std::os but now have non-thread-blocking
implementations as part of using the current IoFactory.
The api of io::file is in flux, but I plan on changing it in the next commit as
well.
Closes#10057
The invocation for making a directory should be able to specify a mode to make
the directory with (instead of defaulting to one particular mode). Additionally,
libuv and various OSes implement efficient versions of renaming files, so this
operation is exposed as an IoFactory call.
Now that the type_id intrinsic is working across crates, all of these
unnecessary messages can be removed to have the failure type for a task truly be
~Any and only ~Any
Tests now have the same name as the test that they're running (to allow for
easier diagnosing of failure sources), and the main task is now specially named
`<main>` instead of `<unnamed>`.
Closes#10195Closes#10073
This takes the last reforms on the `Option` type and applies them to `Result` too. For that, I reordered and grouped the functions in both modules, and also did some refactorings:
- Added `as_ref` and `as_mut` adapters to `Result`.
- Renamed `Result::map_move` to `Result::map` (same for `_err` variant), deleted other map functions.
- Made the `.expect()` methods be generic over anything you can
fail with.
- Updated some doc comments to the line doc comment style
- Cleaned up and extended standard trait implementations on `Option` and `Result`
- Removed legacy implementations in the `option` and `result` module
Tests now have the same name as the test that they're running (to allow for
easier diagnosing of failure sources), and the main task is now specially named
<main> instead of <unnamed>.
Closes#10195Closes#10073
The previous method was unsound because you could very easily create two mutable
pointers which alias the same location (not sound behavior). This hides the
function which does so and then exports an explicit flush() function (with
documentation about how it works).
Cleaned up the source in a few places
Renamed `map_move` to `map`, removed other `map` methods
Added `as_ref` and `as_mut` adapters to `Result`
Added `fmt::Default` impl
The previous method was unsound because you could very easily create two mutable
pointers which alias the same location (not sound behavior). This hides the
function which does so and then exports an explicit flush() function (with
documentation about how it works).
- `begin_unwind` and `fail!` is now generic over any `T: Any + Send`.
- Every value you fail with gets boxed as an `~Any`.
- Because of implementation issues, `&'static str` and `~str` are still
handled specially behind the scenes.
- Changed the big macro source string in libsyntax to a raw string
literal, and enabled doc comments there.
- `begin_unwind` is now generic over any `T: Any + Send`.
- Every value you fail with gets boxed as an `~Any`.
- Because of implementation details, `&'static str` and `~str` are still
handled specially behind the scenes.
- Changed the big macro source string in libsyntax to a raw string
literal, and enabled doc comments there.
Allows an enum with a discriminant to use any of the primitive integer types to store it. By default the smallest usable type is chosen, but this can be overridden with an attribute: `#[repr(int)]` etc., or `#[repr(C)]` to match the target's C ABI for the equivalent C enum.
Also adds a lint pass for using non-FFI safe enums in extern declarations, checks that specified discriminants can be stored in the specified type if any, and fixes assorted code that was assuming int.
This is one of the final steps needed to complete #9128. It still needs a little bit of polish before closing that issue, but it's in a pretty much "done" state now.
The idea here is that the entire event loop implementation using libuv is now housed in `librustuv` as a completely separate library. This library is then injected (via `extern mod rustv`) into executable builds (similarly to how libstd is injected, tunable via `#[no_uv]`) to bring in the "rust blessed event loop implementation."
Codegen-wise, there is a new `event_loop_factory` language item which is tagged on a function with 0 arguments returning `~EventLoop`. This function's symbol is then inserted into the crate map for an executable crate, and if there is no definition of the `event_loop_factory` language item then the value is null.
What this means is that embedding rust as a library in another language just got a little harder. Libraries don't have crate maps, which means that there's no way to find the event loop implementation to spin up the runtime. That being said, it's always possible to build the runtime manually. This request also makes more runtime components public which should probably be public anyway. This new public-ness should allow custom scheduler setups everywhere regardless of whether you follow the `rt::start `path.
There are a few reasons that this is a desirable move to take:
1. Proof of concept that a third party event loop is possible
2. Clear separation of responsibility between rt::io and the uv-backend
3. Enforce in the future that the event loop is "pluggable" and replacable
Here's a quick summary of the points of this pull request which make this
possible:
* Two new lang items were introduced: event_loop, and event_loop_factory.
The idea of a "factory" is to define a function which can be called with no
arguments and will return the new event loop as a trait object. This factory
is emitted to the crate map when building an executable. The factory doesn't
have to exist, and when it doesn't then an empty slot is in the crate map and
a basic event loop with no I/O support is provided to the runtime.
* When building an executable, then the rustuv crate will be linked by default
(providing a default implementation of the event loop) via a similar method to
injecting a dependency on libstd. This is currently the only location where
the rustuv crate is ever linked.
* There is a new #[no_uv] attribute (implied by #[no_std]) which denies
implicitly linking to rustuv by default
Closes#5019
Primarily this makes the Scheduler and all of its related interfaces public. The
reason for doing this is that currently any extern event loops had no access to
the scheduler at all. This allows third-party event loops to manipulate the
scheduler, along with allowing the uv event loop to live inside of its own
crate.
This drops more of the old C++ runtime to rather be written in rust. A few
features were lost along the way, but hopefully not too many. The main loss is
that there are no longer backtraces associated with allocations (rust doesn't
have a way of acquiring those just yet). Other than that though, I believe that
the rest of the debugging utilities made their way over into rust.
Closes#8704
Some code cleanup, sorting of import blocks
Removed std::unstable::UnsafeArc's use of Either
Added run-fail tests for the new FailWithCause impls
Changed future_result and try to return Result<(), ~Any>.
- Internally, there is an enum of possible fail messages passend around.
- In case of linked failure or a string message, the ~Any gets
lazyly allocated in future_results recv method.
- For that, future result now returns a wrapper around a Port.
- Moved and renamed task::TaskResult into rt::task::UnwindResult
and made it an internal enum.
- Introduced a replacement typedef `type TaskResult = Result<(), ~Any>`.
I'm not entirely sure why this is happening, but the server task is never seeing
the second send of the client task, and this test will very reliably fail to
complete on windows.
It was pretty much a miracle that these tests were ever passing. They would
never have passed in the single threaded case because only one sigint in the
tests is ever generated, but when run in parallel two sigints will be generated.
This drops more of the old C++ runtime to rather be written in rust. A few
features were lost along the way, but hopefully not too many. The main loss is
that there are no longer backtraces associated with allocations (rust doesn't
have a way of acquiring those just yet). Other than that though, I believe that
the rest of the debugging utilities made their way over into rust.
Closes#8704
This optimizes the `home_for_io` code path by requiring fewer scheduler
operations in some situtations.
When moving to your home scheduler, this no longer forces a context switch if
you're already on the home scheduler. Instead, the homing code now simply pins
you to your current scheduler (making it so you can't be stolen away). If you're
not on your home scheduler, then we context switch away, sending you to your
home scheduler.
When the I/O operation is done, then we also no longer forcibly trigger a
context switch. Instead, the action is cased on whether the task is homed or
not. If a task does not have a home, then the task is re-flagged as not having a
home and no context switch is performed. If a task is homed to the current
scheduler, then we don't do anything, and if the task is homed to a foreign
scheduler, then it's sent along its merry way.
I verified that there are about a third as many `write` syscalls done in print
operations now. Libuv uses write to implement async handles, and the homing
before and after each I/O operation was triggering a write on these async
handles. Additionally, using the terrible benchmark of printing 10k times in a
loop, this drives the runtime from 0.6s down to 0.3s (yay!).
This optimizes the `home_for_io` code path by requiring fewer scheduler
operations in some situtations.
When moving to your home scheduler, this no longer forces a context switch if
you're already on the home scheduler. Instead, the homing code now simply pins
you to your current scheduler (making it so you can't be stolen away). If you're
not on your home scheduler, then we context switch away, sending you to your
home scheduler.
When the I/O operation is done, then we also no longer forcibly trigger a
context switch. Instead, the action is cased on whether the task is homed or
not. If a task does not have a home, then the task is re-flagged as not having a
home and no context switch is performed. If a task is homed to the current
scheduler, then we don't do anything, and if the task is homed to a foreign
scheduler, then it's sent along its merry way.
I verified that there are about a third as many `write` syscalls done in print
operations now. Libuv uses write to implement async handles, and the homing
before and after each I/O operation was triggering a write on these async
handles. Additionally, using the terrible benchmark of printing 10k times in a
loop, this drives the runtime from 0.6s down to 0.3s (yay!).
Almost all languages provide some form of buffering of the stdout stream, and
this commit adds this feature for rust. A handle to stdout is lazily initialized
in the Task structure as a buffered owned Writer trait object. The buffer
behavior depends on where stdout is directed to. Like C, this line-buffers the
stream when the output goes to a terminal (flushes on newlines), and also like C
this uses a fixed-size buffer when output is not directed at a terminal.
We may decide the fixed-size buffering is overkill, but it certainly does reduce
write syscall counts when piping output elsewhere. This is a *huge* benefit to
any code using logging macros or the printing macros. Formatting emits calls to
`write` very frequently, and to have each of them backed by a write syscall was
very expensive.
In a local benchmark of printing 10000 lines of "what" to stdout, I got the
following timings:
when | terminal | redirected
----------|---------------|--------
before | 0.575s | 0.525s
after | 0.197s | 0.013s
C | 0.019s | 0.004s
I can also confirm that we're buffering the output appropriately in both
situtations. We're still far slower than C, but I believe much of that has to do
with the "homing" that all tasks due, we're still performing an order of
magnitude more write syscalls than C does.
Almost all languages provide some form of buffering of the stdout stream, and
this commit adds this feature for rust. A handle to stdout is lazily initialized
in the Task structure as a buffered owned Writer trait object. The buffer
behavior depends on where stdout is directed to. Like C, this line-buffers the
stream when the output goes to a terminal (flushes on newlines), and also like C
this uses a fixed-size buffer when output is not directed at a terminal.
We may decide the fixed-size buffering is overkill, but it certainly does reduce
write syscall counts when piping output elsewhere. This is a *huge* benefit to
any code using logging macros or the printing macros. Formatting emits calls to
`write` very frequently, and to have each of them backed by a write syscall was
very expensive.
In a local benchmark of printing 10000 lines of "what" to stdout, I got the
following timings:
when | terminal | redirected
----------------------------------
before | 0.575s | 0.525s
after | 0.197s | 0.013s
C | 0.019s | 0.004s
I can also confirm that we're buffering the output appropriately in both
situtations. We're still far slower than C, but I believe much of that has to do
with the "homing" that all tasks due, we're still performing an order of
magnitude more write syscalls than C does.
It's not guaranteed that there will always be an event loop to run, and this
implementation will serve as an incredibly basic one which does not provide any
I/O, but allows the scheduler to still run.
cc #9128
This is a peculiar function to require event loops to implement, and it's only
used in one spot during tests right now. Instead, a possibly more robust apis
for timers should be used rather than requiring all event loops to implement a
curious-looking function.
The PausibleIdleCallback must have some handle into the event loop, and because
struct destructors are run in order of top-to-bottom in order of fields, this
meant that the event loop was getting destroyed before the idle callback was
getting destroyed.
I can't confirm that this fixes a problem in how we use libuv, but it does
semantically fix a problem for usage with other event loops.
This adds constructors to pipe streams in the new runtime to take ownership of
file descriptors, and also fixes a few tests relating to the std::run changes
(new errors are raised on io_error and one test is xfail'd).
I was seeing a lot of weird behavior with stdin behaving as a tty, and it
doesn't really quite make sense, so instead this moves to using libuv's pipes
instead (which make more sense for stdin specifically).
This prevents piping input to rustc hanging forever.
The general idea is to remove conditions completely from I/O, so in the meantime
remove the read_error condition to mean the same thing as the io_error condition.
The isn't an ideal patch, and the comment why is in the code. Basically uvio
uses task::unkillable which touches the kill flag for a task, and if the task is
failing due to mismangement of the kill flag, then there will be serious
problems when the task tries to print that it's failing.
When uv's TTY I/O is used for the stdio streams, the file descriptors are put
into a non-blocking mode. This means that other concurrent writes to the same
stream can fail with EAGAIN or EWOULDBLOCK. By all I/O to event-loop I/O, we
avoid this error.
There is one location which cannot move, which is the runtime's dumb_println
function. This was implemented to handle the EAGAIN and EWOULDBLOCK errors and
simply retry again and again.
This involved changing a fair amount of code, rooted in how we access the local
IoFactory instance. I added a helper method to the rtio module to access the
optional local IoFactory. This is different than before in which it was assumed
that a local IoFactory was *always* present. Now, a separate io_error is raised
when an IoFactory is not present, yet I/O is requested.
This removes the PathLike trait associated with this "support module". This is
yet another "container of bytes" trait, so I didn't want to duplicate what
already exists throughout libstd. In actuality, we're going to pass of C strings
to the libuv APIs, so instead the arguments are now bound with the 'ToCStr'
trait instead.
Additionally, a layer of complexity was removed by immediately converting these
type-generic parameters into CStrings to get handed off to libuv apis.
We get a little more functionality from libuv for these kinds of streams (things
like terminal dimentions), and it also appears to more gracefully handle the
stream being a window. Beforehand, if you used stdio and hit CTRL+d on a
process, libuv would continually return 0-length successful reads instead of
interpreting that the stream was closed.
I was hoping to be able to write tests for this, but currently the testing
infrastructure doesn't allow tests with a stdin and a stdout, but this has been
manually tested! (not that it means much)
