This optimizes the `home_for_io` code path by requiring fewer scheduler
operations in some situtations.
When moving to your home scheduler, this no longer forces a context switch if
you're already on the home scheduler. Instead, the homing code now simply pins
you to your current scheduler (making it so you can't be stolen away). If you're
not on your home scheduler, then we context switch away, sending you to your
home scheduler.
When the I/O operation is done, then we also no longer forcibly trigger a
context switch. Instead, the action is cased on whether the task is homed or
not. If a task does not have a home, then the task is re-flagged as not having a
home and no context switch is performed. If a task is homed to the current
scheduler, then we don't do anything, and if the task is homed to a foreign
scheduler, then it's sent along its merry way.
I verified that there are about a third as many `write` syscalls done in print
operations now. Libuv uses write to implement async handles, and the homing
before and after each I/O operation was triggering a write on these async
handles. Additionally, using the terrible benchmark of printing 10k times in a
loop, this drives the runtime from 0.6s down to 0.3s (yay!).
This optimizes the `home_for_io` code path by requiring fewer scheduler
operations in some situtations.
When moving to your home scheduler, this no longer forces a context switch if
you're already on the home scheduler. Instead, the homing code now simply pins
you to your current scheduler (making it so you can't be stolen away). If you're
not on your home scheduler, then we context switch away, sending you to your
home scheduler.
When the I/O operation is done, then we also no longer forcibly trigger a
context switch. Instead, the action is cased on whether the task is homed or
not. If a task does not have a home, then the task is re-flagged as not having a
home and no context switch is performed. If a task is homed to the current
scheduler, then we don't do anything, and if the task is homed to a foreign
scheduler, then it's sent along its merry way.
I verified that there are about a third as many `write` syscalls done in print
operations now. Libuv uses write to implement async handles, and the homing
before and after each I/O operation was triggering a write on these async
handles. Additionally, using the terrible benchmark of printing 10k times in a
loop, this drives the runtime from 0.6s down to 0.3s (yay!).
Almost all languages provide some form of buffering of the stdout stream, and
this commit adds this feature for rust. A handle to stdout is lazily initialized
in the Task structure as a buffered owned Writer trait object. The buffer
behavior depends on where stdout is directed to. Like C, this line-buffers the
stream when the output goes to a terminal (flushes on newlines), and also like C
this uses a fixed-size buffer when output is not directed at a terminal.
We may decide the fixed-size buffering is overkill, but it certainly does reduce
write syscall counts when piping output elsewhere. This is a *huge* benefit to
any code using logging macros or the printing macros. Formatting emits calls to
`write` very frequently, and to have each of them backed by a write syscall was
very expensive.
In a local benchmark of printing 10000 lines of "what" to stdout, I got the
following timings:
when | terminal | redirected
----------|---------------|--------
before | 0.575s | 0.525s
after | 0.197s | 0.013s
C | 0.019s | 0.004s
I can also confirm that we're buffering the output appropriately in both
situtations. We're still far slower than C, but I believe much of that has to do
with the "homing" that all tasks due, we're still performing an order of
magnitude more write syscalls than C does.
Almost all languages provide some form of buffering of the stdout stream, and
this commit adds this feature for rust. A handle to stdout is lazily initialized
in the Task structure as a buffered owned Writer trait object. The buffer
behavior depends on where stdout is directed to. Like C, this line-buffers the
stream when the output goes to a terminal (flushes on newlines), and also like C
this uses a fixed-size buffer when output is not directed at a terminal.
We may decide the fixed-size buffering is overkill, but it certainly does reduce
write syscall counts when piping output elsewhere. This is a *huge* benefit to
any code using logging macros or the printing macros. Formatting emits calls to
`write` very frequently, and to have each of them backed by a write syscall was
very expensive.
In a local benchmark of printing 10000 lines of "what" to stdout, I got the
following timings:
when | terminal | redirected
----------------------------------
before | 0.575s | 0.525s
after | 0.197s | 0.013s
C | 0.019s | 0.004s
I can also confirm that we're buffering the output appropriately in both
situtations. We're still far slower than C, but I believe much of that has to do
with the "homing" that all tasks due, we're still performing an order of
magnitude more write syscalls than C does.
It's not guaranteed that there will always be an event loop to run, and this
implementation will serve as an incredibly basic one which does not provide any
I/O, but allows the scheduler to still run.
cc #9128
This is a peculiar function to require event loops to implement, and it's only
used in one spot during tests right now. Instead, a possibly more robust apis
for timers should be used rather than requiring all event loops to implement a
curious-looking function.
The PausibleIdleCallback must have some handle into the event loop, and because
struct destructors are run in order of top-to-bottom in order of fields, this
meant that the event loop was getting destroyed before the idle callback was
getting destroyed.
I can't confirm that this fixes a problem in how we use libuv, but it does
semantically fix a problem for usage with other event loops.
This adds constructors to pipe streams in the new runtime to take ownership of
file descriptors, and also fixes a few tests relating to the std::run changes
(new errors are raised on io_error and one test is xfail'd).
I was seeing a lot of weird behavior with stdin behaving as a tty, and it
doesn't really quite make sense, so instead this moves to using libuv's pipes
instead (which make more sense for stdin specifically).
This prevents piping input to rustc hanging forever.
The general idea is to remove conditions completely from I/O, so in the meantime
remove the read_error condition to mean the same thing as the io_error condition.
The isn't an ideal patch, and the comment why is in the code. Basically uvio
uses task::unkillable which touches the kill flag for a task, and if the task is
failing due to mismangement of the kill flag, then there will be serious
problems when the task tries to print that it's failing.
When uv's TTY I/O is used for the stdio streams, the file descriptors are put
into a non-blocking mode. This means that other concurrent writes to the same
stream can fail with EAGAIN or EWOULDBLOCK. By all I/O to event-loop I/O, we
avoid this error.
There is one location which cannot move, which is the runtime's dumb_println
function. This was implemented to handle the EAGAIN and EWOULDBLOCK errors and
simply retry again and again.
This involved changing a fair amount of code, rooted in how we access the local
IoFactory instance. I added a helper method to the rtio module to access the
optional local IoFactory. This is different than before in which it was assumed
that a local IoFactory was *always* present. Now, a separate io_error is raised
when an IoFactory is not present, yet I/O is requested.
This removes the PathLike trait associated with this "support module". This is
yet another "container of bytes" trait, so I didn't want to duplicate what
already exists throughout libstd. In actuality, we're going to pass of C strings
to the libuv APIs, so instead the arguments are now bound with the 'ToCStr'
trait instead.
Additionally, a layer of complexity was removed by immediately converting these
type-generic parameters into CStrings to get handed off to libuv apis.
We get a little more functionality from libuv for these kinds of streams (things
like terminal dimentions), and it also appears to more gracefully handle the
stream being a window. Beforehand, if you used stdio and hit CTRL+d on a
process, libuv would continually return 0-length successful reads instead of
interpreting that the stream was closed.
I was hoping to be able to write tests for this, but currently the testing
infrastructure doesn't allow tests with a stdin and a stdout, but this has been
manually tested! (not that it means much)
Removed unused import warning in std::mem and cleaned it up too
Removed is_true and is_false from std::bool
Removed freestanding functions in std::bool
- Adds the `Sample` and `IndependentSample` traits for generating numbers where there are parameters (e.g. a list of elements to draw from, or the mean/variance of a normal distribution). The former takes `&mut self` and the latter takes `&self` (this is the only difference).
- Adds proper `Normal` and `Exp`-onential distributions
- Adds `Range` which generates `[lo, hi)` generically & properly (via a new trait) replacing the incorrect behaviour of `Rng.gen_integer_range` (this has become `Rng.gen_range` for convenience, it's far more efficient to use `Range` itself)
- Move the `Weighted` struct from `std::rand` to `std::rand::distributions` & improve it
- optimisations and docs
- Use ["nothing up my sleeve numbers"](http://en.wikipedia.org/wiki/Nothing_up_my_sleeve_number) for the ISAAC tests.
- Replace the default implementation of `Rng.fill_bytes` with something that doesn't try to do bad things with `transmute` and vectors just for the sake of a little speed.
- Replace the transmutes used to seed the ISAAC RNGs with calls into `vec::raw`.
Slice transmutes are now (and, really, always were) dangerous, so we
avoid them and do the (only?) non-(undefined behaviour in C) pointer
cast: casting to *u8.
I'm planning on doing more updates, but the section in the tutorial stood out at me since the 'rust' tool no longer exists, this should probably be removed to lessen confusion.
This reifies the computations required for uniformity done by
(the old) `Rng.gen_integer_range` (now Rng.gen_range), so that they can
be amortised over many invocations, if it is called in a loop.
Also, it makes it correct, but using a trait + impls for each type,
rather than trying to coerce `Int` + `u64` to do the right thing. This
also makes it more extensible, e.g. big integers could & should
implement SampleRange.
Complete the implementation of Exp and Normal started by Exp1 and
StandardNormal by creating types implementing Sample & IndependentSample
with the appropriate parameters.
This should close#9468.
I removed the test stating that nested comments should not be implemented.
I had a little chicken-and-egg problem because a comment of the std contains "/*", and adding support for nested comment creates a backward incompatibility in that case, so I had to use a dirty hack to get stage1 and stage2 to compile. This part should be revert when this commit lands in a snapshot.
This is my first non-typo contribution, so I'm open to any comment.
This commit re-introduces the functionality of __morestack in a way that it was
not originally anticipated. Rust does not currently have segmented stacks,
rather just large stack segments. We do not detect when these stack segments are
overrun currently, but this commit leverages __morestack in order to check this.
This commit purges a lot of the old __morestack and stack limit C++
functionality, migrating the necessary chunks to rust. The stack limit is now
entirely maintained in rust, and the "main logic bits" of __morestack are now
also implemented in rust as well.
I put my best effort into validating that this currently builds and runs successfully on osx and linux 32/64 bit, but I was unable to get this working on windows. We never did have unwinding through __morestack frames, and although I tried poking at it for a bit, I was unable to understand why we don't get unwinding right now.
A focus of this commit is to implement as much of the logic in rust as possible. This involved some liberal usage of `no_split_stack` in various locations, along with some use of the `asm!` macro (scary). I modified a bit of C++ to stop calling `record_sp_limit` because this is no longer defined in C++, rather in rust.
Another consequence of this commit is that `thread_local_storage::{get, set}` must both be flagged with `#[rust_stack]`. I've briefly looked at the implementations on osx/linux/windows to ensure that they're pretty small stacks, and I'm pretty sure that they're definitely less than 20K stacks, so we probably don't have a lot to worry about.
Other things worthy of note:
* The default stack size is now 4MB instead of 2MB. This is so that when we request 2MB to call a C function you don't immediately overflow because you have consumed any stack at all.
* `asm!` is actually pretty cool, maybe we could actually define context switching with it?
* I wanted to add links to the internet about all this jazz of storing information in TLS, but I was only able to find a link for the windows implementation. Otherwise my suggestion is just "disassemble on that arch and see what happens"
* I put my best effort forward on arm/mips to tweak __morestack correctly, we have no ability to test this so an extra set of eyes would be useful on these spots.
* This is all really tricky stuff, so I tried to put as many comments as I thought were necessary, but if anything is still unclear (or I completely forgot to take something into account), I'm willing to write more!
This commit resumes management of the stack boundaries and limits when switching
between tasks. This additionally leverages the __morestack function to run code
on "stack overflow". The current behavior is to abort the process, but this is
probably not the best behavior in the long term (for deails, see the comment I
wrote up in the stack exhaustion routine).
d4a32386f3 broke these since slice_to() and slice_from() must get character
boundaries, and arbitrary needle lengths don't necessarily map to character
boundaries of the haystack.
This also adds new tests that would have caught this bug.
d4a32386f3 broke these since slice_to() and slice_from() must get character
boundaries, and arbitrary needle lengths don't necessarily map to character
boundaries of the haystack.
This also adds new tests that would have caught this bug.
Hello,
First time rust contributor here, please let me know if I need to sort out the contribution agreement for this.
I picked issue #9755 to dip my toe in the water, this pull request isn't quite complete though as I have not updated the documentation. The reason for this is that I haven't tracked down why this feature is gated so I don't feel I can write a justification of the same quality as the other features have been documented.
If someone would like to explain or point me at a mail thread I am happy to update with this change.
Hopefully I have understood the process of converting the old flag into a directive correctly.
Also just to call out what I am sure if a known quirk when adding feature directives, you can't build this code unless you have a snapshot of the compiler which knows about the feature directive. Chicken and the egg. I split the change into two commits, the first should be able to build a snapshot that can compile the second.
std::iter: Introduce .by_ref() adaptor
Creates a wrapper around a mutable reference to the iterator.
This is useful to allow applying iterator adaptors while still
retaining ownership of the original iterator value.
Example::
let mut xs = range(0, 10);
// sum the first five values
let partial_sum = xs.by_ref().take(5).fold(0, |a, b| a + b);
assert!(partial_sum == 10);
// xs.next() is now `5`
assert!(xs.next() == Some(5));
---
This adaptor requires the user to have good understanding of
iterators or what a particular adaptor does. There could be some
pitfalls here with the iterator protocol, it's mostly the same issues
as other places regarding what happens after the iterator
returns None for the first time.
There could also be other ways to achieve the same thing, for
example Implementing iterator on `&mut T` itself:
`impl <T: Iterator<..>> Iterator for &mut T` but that would only
lead to confusion I think.
The goal here is to avoid requiring a division or multiplication to compare against the length. The bounds check previously used an incorrect micro-optimization to replace the division by a multiplication, but now neither is necessary *for slices*. Unique/managed vectors will have to do a division to get the length until they are reworked/replaced.
Rewrite the entire `std::path` module from scratch.
`PosixPath` is now based on `~[u8]`, which fixes#7225.
Unnecessary allocation has been eliminated.
There are a lot of clients of `Path` that still assume utf-8 paths.
This is covered in #9639.
Delete the following API functions:
- set_dirname()
- with_dirname()
- set_filestem()
- with_filestem()
- add_extension()
- file_path()
Also change pop() to return a boolean instead of an owned copy of the
old filename.
Standardize the is_sep() functions to be the same in both posix and
windows, and re-export from path. Update extra::glob to use this.
Remove the usage of either, as it's going away.
Move the WindowsPath-specific methods out of WindowsPath and make them
top-level functions of path::windows instead. This way you cannot
accidentally write code that will fail to compile on non-windows
architectures without typing ::windows anywhere.
Remove GenericPath::from_c_str() and just impl BytesContainer for
CString instead.
Remove .join_path() and .push_path() and just implement BytesContainer
for Path instead.
Remove FilenameDisplay and add a boolean flag to Display instead.
Remove .each_parent(). It only had one caller, so just inline its
definition there.
...al work
This is causing really awful scheduler behavior where the main thread scheduler is
continually waking up, stealing work, discovering it can't actually run the work,
and sending it off to another scheduler.
No test cases because we don't have suitable instrumentation for it.
* Allow named parameters to specify width/precision
* Intepret the format string '0$' as "width is the 0th argument" instead of
thinking the lone '0' was the sign-aware-zero-padding flag. To get both you'd
need to put '00$' which makes more sense if you want both to happen.
Closes#9669
Rewrite these methods as methods on Display and FilenameDisplay. This
turns
do path.with_display_str |s| { ... }
into
do path.display().with_str |s| { ... }
Add a new trait BytesContainer that is implemented for both byte vectors
and strings.
Convert Path::from_vec and ::from_str to one function, Path::new().
Remove all the _str-suffixed mutation methods (push, join, with_*,
set_*) and modify the non-suffixed versions to use BytesContainer.
Remove the old path.
Rename path2 to path.
Update all clients for the new path.
Also make some miscellaneous changes to the Path APIs to help the
adoption process.
These methods return an object that can be formatted using {} to print
display strings.
Path itself does not implement fmt::Default to avoid accidental usage of
display strings in incorrect places (e.g. process arguments).
These functions are for working with a string representation of the path
even if it's not UTF-8 encoded. They replace invalid UTF-8 sequences
with the replacement char.
As documented in #7225, we cannot rely on paths being representable in
utf-8. Specifically, Linux allows anything (besides NUL) in a path.
Redesign GenericPath in light of this.
PosixPath hasn't been reimplemented yet for ~[u8].
This is causing really awful scheduler behavior where the main thread scheduler is
continually waking up, stealing work, discovering it can't actually run the work,
and sending it off to another scheduler.
This adds `get_opt` to `std::vec`, which looks up an item by index and returns an `Option`. If the given index is out of range, `None` will be returned, otherwise a `Some`-wrapped item will be returned.
Example use case:
```rust
use std::os;
fn say_hello(name: &str) {
println(fmt!("Hello, %s", name));
}
fn main(){
// Try to get the first cmd line arg, but default to "World"
let args = os::args();
let default = ~"World";
say_hello(*args.get_opt(1).unwrap_or(&default));
}
```
If there's an existing way of implementing this pattern that's cleaner, I'll happily close this. I'm also open to naming suggestions (`index_opt`?)
This patch removes the code responsible for handling older CrateMap versions (as discussed during #9593). Only the new (safer) layout is supported now.
I've left out a way to construct from a `Send` type until #9509 is resolved. I am confident that this interface can remain backwards compatible though, assuming we name the `Pointer` trait method `borrow`.
When there is a way to convert from `Send` (`from_send`), a future RAII-based `Mut` type can be used with this to implemented a mutable reference-counted pointer. For now, I've left around the `RcMut` type but it may drastically change or be removed.
This implements a number of the baby steps needed to start eliminating everything inside of `std::io`. It turns out that there are a *lot* of users of that module, so I'm going to try to tackle them separately instead of bringing down the whole system all at once.
This pull implements a large amount of unimplemented functionality inside of `std::rt::io` including:
* Native file I/O (file descriptors, *FILE)
* Native stdio (through the native file descriptors)
* Native processes (extracted from `std::run`)
I also found that there are a number of users of `std::io` which desire to read an input line-by-line, so I added an implementation of `read_until` and `read_line` to `BufferedReader`.
With all of these changes in place, I started to axe various usages of `std::io`. There's a lot of one-off uses here-and-there, but the major use-case remaining that doesn't have a fantastic solution is `extra::json`. I ran into a few compiler bugs when attempting to remove that, so I figured I'd come back to it later instead.
There is one fairly major change in this pull, and it's moving from native stdio to uv stdio via `print` and `println`. Unfortunately logging still goes through native I/O (via `dumb_println`). This is going to need some thinking, because I still want the goal of logging/printing to be 0 allocations, and this is not possible if `io::stdio::stderr()` is called on each log message. Instead I think that this may need to be cached as the `logger` field inside the `Task` struct, but that will require a little more workings to get right (this is also a similar problem for print/println, do we cache `stdout()` to not have to re-create it every time?).
The minimum (negative) value of a float is `-Bounded::max_value()`, not `Bounded::min_value()`.
Otherwise the following has an incorrect behavior:
```rust
let a = -1.0f64;
let b: f32 = NumCast::from(a); // incorrectly returns None
```
This changes an `assert_once_ever!` assertion to just a plain old assertion
around an atomic boolean to ensure that one particular runtime doesn't attempt
to exit twice.
Closes#9739
This changes an `assert_once_ever!` assertion to just a plain old assertion
around an atomic boolean to ensure that one particular runtime doesn't attempt
to exit twice.
Closes#9739
This lets the C++ code in the rt handle the (slightly) tricky parts of
random number generation: e.g. error detection/handling, and using the
values of the `#define`d options to the various functions.
It now:
- can be explicitly seeded from user code (`seed_task_rng`) or from the
environment (`RUST_SEED`, a positive integer)
- automatically reseeds itself from the OS *unless* it was seeded by
either method above
- has more documentation
This provides 2 methods: .reseed() and ::from_seed that modify and
create respecitively.
Implement this trait for the RNGs in the stdlib for which this makes
sense.
This is implemented as a wrapper around another RNG. It is designed
to allow the actual implementation to be changed without changing
the external API (e.g. it currently uses a 64-bit generator on 64-
bit platforms, and a 32-bit one on 32-bit platforms; but one could
imagine that the IsaacRng may be deprecated later, and having this
ability to switch algorithms without having to update the points of
use is convenient.)
This is the recommended general use RNG.
The former reads from e.g. /dev/urandom, the latter just wraps any
std::rt::io::Reader into an interface that implements Rng.
This also adds Rng.fill_bytes for efficient implementations of the above
(reading 8 bytes at a time is inefficient when you can read 1000), and
removes the dependence on src/rt (i.e. rand_gen_seed) although this last
one requires implementing hand-seeding of the XorShiftRng used in the
scheduler on Linux/unixes, since OSRng relies on a scheduler existing to
be able to read from /dev/urandom.
This is 2x faster on 64-bit computers at generating anything larger
than 32-bits.
It has been verified against the canonical C implementation from the
website of the creator of ISAAC64.
Also, move `Rng.next` to `Rng.next_u32` and add `Rng.next_u64` to
take full advantage of the wider word width; otherwise Isaac64 will
always be squeezed down into a u32 wasting half the entropy and
offering no advantage over the 32-bit variant.
This commit fixes all of the fallout of the previous commit which is an attempt
to refine privacy. There were a few unfortunate leaks which now must be plugged,
and the most horrible one is the current `shouldnt_be_public` module now inside
`std::rt`. I think that this either needs a slight reorganization of the
runtime, or otherwise it needs to just wait for the external users of these
modules to get replaced with their `rt` implementations.
Other fixes involve making things pub which should be pub, and otherwise
updating error messages that now reference privacy instead of referencing an
"unresolved name" (yay!).
This implements the necessary logic for gating particular features off by default in the compiler. There are a number of issues which have been wanting this form of mechanism, and this initially gates features which we have open issues for.
Additionally, this should unblock #9255
This pull request changes to memory layout of the `CrateMap` struct to use static slices instead of raw pointers. Most of the discussion took place [here](63b5975efa (L1R92)) .
The memory layout of CrateMap changed, without bumping the version number in the struct. Another, more backward compatible, solution would be to keep the old code and increase the version number in the new struct. On the other hand, the `annihilate_fn` pointer was removed without bumping the version number recently.
At the moment, the stage0 compiler does not use the new memory layout, which would lead the segfaults during stage0 compilation, so I've added a dummy `iter_crate_map` function for stage0, which does nothing. Again, this could be avoided if we'd bump the version number in the struct and keep the old code.
I'd like to use a normal `for` loop [here](https://github.com/fhahn/rust/compare/logging-unsafe-removal?expand=1#L1R109),
for child in children.iter() {
do_iter_crate_map(child, |x| f(x), visited);
}
but for some reason this only yields `error: unresolved enum variant, struct or const 'Some'` and I have no idea why.
A few features are now hidden behind various #[feature(...)] directives. These
include struct-like enum variants, glob imports, and macro_rules! invocations.
Closes#9304Closes#9305Closes#9306Closes#9331
The root issue is that dlerror isn't reentrant or even thread safe.
The solution implemented here is to make a yielding spin lock over an
AtomicFlag. This is pretty hacky, but the best we can do at this point.
As far as I can tell, it isn't possible to create a global mutex without
having to initialize it in a single threaded context.
The Windows code isn't affected since errno is thread-local on Windows
and it's running in an atomically block to ensure there isn't a green
thread context switch.
Closes#8156
This PR solves one of the pain points with c-style enums. Simplifies writing a fn to convert from an int/uint to an enum. It does this through a `#[deriving(FromPrimitive)]` syntax extension.
Before this is committed though, we need to discuss if `ToPrimitive`/`FromPrimitive` has the right design (cc #4819). I've changed all the `.to_int()` and `from_int()` style functions to return `Option<int>` so we can handle partial functions. For this PR though only enums and `extra::num::bigint::*` take advantage of returning None for unrepresentable values. In the long run it'd be better if `i64.to_i8()` returned `None` if the value was too large, but I'll save this for a future PR.
Closes#3868.
The root issue is that dlerror isn't reentrant or even thread safe.
The Windows code isn't affected since errno is thread-local on Windows
and it's running in an atomically block to ensure there isn't a green
thread context switch.
Closes#8156
