I found these automatically, but fixed them manually to ensure the semantics are correct. I know things like these are hardly important, since they only marginally improve clarity. But at least for me typos and simple grammatical errors trigger an---unjustified---sense of unprofessionalism, despite the fact that I make them all the time and I understand that they're the sort of thing that is bound to slip through review.
Anyway, to find most of these I used:
* `ag '.*//.*(\b[A-Za-z]{2,}\b) \1\b'` for repeated words
* `ag '\b(the|this|those|these|a|it) (a|the|this|those|these|it)\b'` to find constructs like 'the this' etc. many false positives, but not too hard to scroll through them to actually find the mistakes.
* `cat ../../typos.txt | paste -d'|' - - - - - - - - - - - - - - - - - - - - - - | tr '\n' '\0' | xargs -0 -P4 -n1 ag`. Hacky way to find misspellings, but it works ok. I got `typos.txt` from [Wikipedia](https://en.wikipedia.org/wiki/Wikipedia:Lists_of_common_misspellings/For_machines)
* `ag '.*//.* a ([ae][a-z]|(o[^n])|(i[a-rt-z]))'` to find places where 'a' was followed by a vowel (requiring 'an' instead).
I also used a handful more one off regexes that are too boring to reproduce here.
Move private bignum module to core::num, because it is not only used in flt2dec.
Extract private 80-bit soft-float into new core::num module for the same reason.
Move private bignum module to core::num, because it is not only used in flt2dec.
Extract private 80-bit soft-float into new core::num module for the same reason.
Do so by using the fact that fixed size arrays (like `[u8; 8]` can be coerced
to slices `&[u8]`, this is expressed through the trait `Unsize<[T]>` that all
fixed size arrays implement.
* Rename `Utf16Items` to `Utf16Decoder`. "Items" is meaningless.
* Generalize it to any `u16` iterator, not just `[u16].iter()`
* Make it yield `Result` instead of a custom `Utf16Item` enum that was isomorphic to `Result`. This enable using the `FromIterator for Result` impl.
* Replace `Utf16Item::to_char_lossy` with a `Utf16Decoder::lossy` iterator adaptor.
This is a [breaking change], but only for users of the unstable `rustc_unicode` crate.
I’d like this functionality to be stabilized and re-exported in `std` eventually, as the "low-level equivalent" of `String::from_utf16` and `String::from_utf16_lossy` like #27784 is the low-level equivalent of #27714.
CC @aturon, @alexcrichton
The iterator protocol specifies that the iteration ends with the return
value `None` from `.next()` (or `.next_back()`) and it is unspecified
what further calls return. The chain adaptor must account for this in
its DoubleEndedIterator implementation.
It uses three states:
- Both `a` and `b` are valid
- Only the Front iterator (`a`) is valid
- Only the Back iterator (`b`) is valid
The fourth state (neither iterator is valid) only occurs after Chain has
returned None once, so we don't need to store this state.
Fixes#26316
* Rename `utf16_items` to `decode_utf16`. "Items" is meaningless.
* Move it to `rustc_unicode::char`, exposed in `std::char`.
* Generalize it to any `u16` iterable, not just `&[u16]`.
* Make it yield `Result` instead of a custom `Utf16Item` enum that was isomorphic to `Result`. This enable using the `FromIterator for Result` impl.
* Add a `REPLACEMENT_CHARACTER` constant.
* Document how `result.unwrap_or(REPLACEMENT_CHARACTER)` replaces `Utf16Item::to_char_lossy`.
These commits move libcore into a state so that it's ready for stabilization, performing some minor cleanup:
* The primitive modules for integers in the standard library were all removed from the source tree as they were just straight reexports of the libcore variants.
* The `core::atomic` module now lives in `core::sync::atomic`. The `core::sync` module is otherwise empty, but ripe for expansion!
* The `core::prelude::v1` module was stabilized after auditing that it is a subset of the standard library's prelude plus some primitive extension traits (char, str, and slice)
* Some unstable-hacks for float parsing errors were shifted around to not use the same unstable hacks (e.g. the `flt2dec` module is now used for "privacy").
After this commit, the remaining large unstable functionality specific to libcore is:
* `raw`, `intrinsics`, `nonzero`, `array`, `panicking`, `simd` -- these modules are all unstable or not reexported in the standard library, so they're just remaining in the same status quo as before
* `num::Float` - this extension trait for floats needs to be audited for functionality (much of that is happening in #27823) and may also want to be renamed to `FloatExt` or `F32Ext`/`F64Ext`.
* Should the extension traits for primitives be stabilized in libcore?
I believe other unstable pieces are not isolated to just libcore but also affect the standard library.
cc #27701
There wasn't any particular reason the functions needed to be there
anyway, so just get rid of them, and adjust libstd to compensate.
With this change, libcore depends on exactly two floating-point functions:
fmod and fmodf. They are implicitly referenced because they are used to
implement "%".
Dependencies of libcore on Linux x86-x64 with this patch:
```
0000000000000000 *UND* 0000000000000000 __powidf2
0000000000000000 *UND* 0000000000000000 __powisf2
0000000000000000 *UND* 0000000000000000 fmod
0000000000000000 *UND* 0000000000000000 fmodf
0000000000000000 *UND* 0000000000000000 memcmp
0000000000000000 *UND* 0000000000000000 memcpy
0000000000000000 *UND* 0000000000000000 memset
0000000000000000 *UND* 0000000000000000 rust_begin_unwind
0000000000000000 *UND* 0000000000000000 rust_eh_personality
```
Stop using stability to hide the implementation details of ParseFloatError and
instead move the error type into the `dec2flt` module. Also move the
implementation blocks of `FromStr for f{32,64}` into `dec2flt` directly.
All of the modules in the standard library were just straight reexports of those
in libcore, so remove all the "macro modules" from the standard library and just
reexport what's in core directly.
There wasn't any particular reason the functions needed to be there
anyway, so just get rid of them, and adjust libstd to compensate.
With this change, libcore depends on exactly two floating-point functions:
fmod and fmodf. They are implicitly referenced because they are
used to implement "%".
* An apparent bug in VS 2013's implementation of the `exp2` function is worked
around in one of flt2dec's tests.
Turns out this was the only fix necessary!
This commit removes all unstable and deprecated functions in the standard
library. A release was recently cut (1.3) which makes this a good time for some
spring cleaning of the deprecated functions.
Completely rewrite the conversion of decimal strings to `f64` and `f32`. The code is intended to be absolutely positively completely 100% accurate (when it doesn't give up). To the best of my knowledge, it achieves that goal. Any input that is not rejected is converted to the floating point number that is closest to the true value of the input. This includes overflow, subnormal numbers, and underflow to zero. In other words, the rounding error is less than or equal to 0.5 units in the last place. Half-way cases (exactly 0.5 ULP error) are handled with half-to-even rounding, also known as banker's rounding.
This code implements the algorithms from the paper [How to Read Floating Point Numbers Accurately][paper] by William D. Clinger, with extensions to handle underflow, overflow and subnormals, as well as some algorithmic optimizations.
# Correctness
With such a large amount of tricky code, many bugs are to be expected. Indeed tracking down the obscure causes of various rounding errors accounts for the bulk of the development time. Extensive tests (taking in the order of hours to run through to completion) are included in `src/etc/test-float-parse`: Though exhaustively testing all possible inputs is impossible, I've had good success with generating millions of instances from various "classes" of inputs. These tests take far too long to be run by @bors so contributors who touch this code need the discipline to run them. There are `#[test]`s, but they don't even cover every stupid mistake I made in course of writing this.
Another aspect is *integer* overflow. Extreme (or malicious) inputs could cause overflow both in the machine-sized integers used for bookkeeping throughout the algorithms (e.g., the decimal exponent) as well as the arbitrary-precision arithmetic. There is input validation to reject all such cases I know of, and I am quite sure nobody will *accidentally* cause this code to go out of range. Still, no guarantees.
# Limitations
Noticed the weasel words "(when it doesn't give up)" at the beginning? Some otherwise well-formed decimal strings are rejected because spelling out the value of the input requires too many digits, i.e., `digits * 10^abs(exp)` can't be stored in a bignum. This only applies if the value is not "obviously" zero or infinite, i.e., if you take a near-infinity or near-zero value and add many pointless fractional digits. At least with the algorithm used here, computing the precise value would require computing the full value as a fraction, which would overflow. The precise limit is `number_of_digits + abs(exp) > 375` but could be raised almost arbitrarily. In the future, another algorithm might lift this restriction entirely.
This should not be an issue for any realistic inputs. Still, the code does reject inputs that would result in a finite float when evaluated with unlimited precision. Some of these inputs are even regressions that the old code (mostly) handled, such as `0.333...333` with 400+ `3`s. Thus this might qualify as [breaking-change].
# Performance
Benchmarks results are... tolerable. Short numbers that hit the fast paths (`f64` multiplication or shortcuts to zero/inf) have performance in the same order of magnitude as the old code tens of nanoseconds. Numbers that are delegated to Algorithm Bellerophon (using floats with 64 bit significand, implemented in software) are slower, but not drastically so (couple hundred nanoseconds).
Numbers that need the AlgorithmM fallback (for `f64`, roughly everything below 1e-305 and above 1e305) take far, far longer, hundreds of microseconds. Note that my implementation is not quite as naive as the expository version in the paper (it needs one to four division instead of ~1000), but division is fundamentally pretty expensive and my implementation of it is extremely simple and slow.
All benchmarks run on a mediocre laptop with a i5-4200U CPU under light load.
# Binary size
Unfortunately the implementation needs to duplicate almost all code: Once for `f32` and once for `f64`. Before you ask, no, this cannot be avoided, at least not completely (but see the Future Work section). There's also a precomputed table of powers of ten, weighing in at about six kilobytes.
Running a stage1 `rustc` over a stand-alone program that simply parses pi to `f32` and `f64` and outputs both results reveals that the overhead vs. the old parsing code is about 44 KiB normally and about 28 KiB with LTO. It's presumably half of that + 3 KiB when only one of the two code paths is exercised.
| rustc options | old | new | delta |
|--------------------------- |--------- |--------- |----------- |
| [nothing] | 2588375 | 2633828 | 44.39 KiB |
| -O | 2585211 | 2630688 | 44.41 KiB |
| -O -C lto | 1026353 | 1054981 | 27.96 KiB |
| -O -C lto -C link-args=-s | 414208 | 442368 | 27.5 KiB |
# Future Work
## Directory layout
The `dec2flt` code uses some types embedded deeply in the `flt2dec` module hierarchy, even though nothing about them it formatting-specific. They should be moved to a more conversion-direction-agnostic location at some point.
## Performance
It could be much better, especially for large inputs. Some low-hanging fruit has been picked but much more work could be done. Some specific ideas are jotted down in `FIXME`s all over the code.
## Binary size
One could try to compress the table further, though I am skeptical. Another avenue would be reducing the code duplication from basically everything being generic over `T: RawFloat`. Perhaps one can reduce the magnitude of the duplication by pushing the parts that don't need to know the target type into separate functions, but this is finicky and probably makes some code read less naturally.
## Other bases
This PR leaves `f{32,64}::from_str_radix` alone. It only replaces `FromStr` (and thus `.parse()`). I am convinced that `from_str_radix` should not exist, and have proposed its [deprecation and speedy removal][deprecate-radix]. Whatever the outcome of that discussion, it is independent from, and out of scope for, this PR.
Fixes#24557Fixes#14353
r? @pnkfelix
cc @lifthrasiir @huonw
[paper]: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.45.4152
[deprecate-radix]: https://internals.rust-lang.org/t/deprecate-f-32-64-from-str-radix/2405
Provides a custom implementation of Iterator methods `count`, `nth`, and `last` for the structures `slice::{Windows,Chunks,ChunksMut}` in the core module.
These implementations run in constant time as opposed to the default implementations which run in linear time.
Addresses Issue #24214
r? @aturon
This commit removes all unstable and deprecated functions in the standard
library. A release was recently cut (1.3) which makes this a good time for some
spring cleaning of the deprecated functions.
Implemented count, nth, and last in constant time for Windows, Chunks,
and ChunksMut created from a slice.
Included checks for overflow in the implementation of nth().
Also added a test for each implemented method to libcoretest.
Addresses #24214
The replacements are functions that usually use a single `mem::transmute` in
their body and restrict input and output via more concrete types than `T` and
`U`. Worth noting are the `transmute` functions for slices and the `from_utf8*`
family for mutable slices. Additionally, `mem::transmute` was often used for
casting raw pointers, when you can already cast raw pointers just fine with
`as`.
This commit primarily adds implementations of the algorithms from William
Clinger's paper "How to Read Floating Point Numbers Accurately". It also
includes a lot of infrastructure necessary for those algorithms, and some
unit tests.
Since these algorithms reject a few (extreme) inputs that were previously
accepted, this could be seen as a [breaking-change]
- Exposing digits and individual bits
- Counting the number of bits
- Add small (digit-sized) values
- Multiplication by power of 5
- Division with remainder
All are necessary for decimal to floating point conversions.
All but the most trivial ones come with tests.
This is necessary for decimal-to-float code (in a later commit) to handle
inputs such as 4.9406564584124654e-324 (the smallest subnormal f64).
According to the benchmarks for flt2dec::dragon, this does not
affect performance measurably. It probably uses slightly more stack
space though.
Improve siphash performance for longer data
Use `ptr::copy_nonoverlapping` (aka memcpy) to load an u64 from the
byte stream. This is correct for any alignment, and the compiler will
use the appropriate instruction to load the data.
Also contains small tweaks that should benefit hashing short data too,
both the commit that removes a variable and the autovectorization of
the hash state initialization (in SipHash::reset).
Benchmarks show that hashing longer data benefits for the improved word loading.
Before (using benchmarks from the first commit in the PR):
The before benchmark is a bit noisy.
```
test hash::sip::bench_bytes_4 ... bench: 41 ns/iter (+/- 0) = 97 MB/s
test hash::sip::bench_bytes_7 ... bench: 49 ns/iter (+/- 2) = 142 MB/s
test hash::sip::bench_bytes_8 ... bench: 42 ns/iter (+/- 4) = 190 MB/s
test hash::sip::bench_bytes_a_16 ... bench: 57 ns/iter (+/- 14) = 280 MB/s
test hash::sip::bench_bytes_b_32 ... bench: 85 ns/iter (+/- 74) = 376 MB/s
test hash::sip::bench_bytes_c_128 ... bench: 278 ns/iter (+/- 33) = 460 MB/s
test hash::sip::bench_long_str ... bench: 825 ns/iter (+/- 103)
test hash::sip::bench_str_of_8_bytes ... bench: 151 ns/iter (+/- 66)
test hash::sip::bench_str_over_8_bytes ... bench: 59 ns/iter (+/- 3)
test hash::sip::bench_str_under_8_bytes ... bench: 47 ns/iter (+/- 56)
test hash::sip::bench_u32 ... bench: 39 ns/iter (+/- 93) = 205 MB/s
test hash::sip::bench_u32_keyed ... bench: 40 ns/iter (+/- 88) = 200 MB/s
test hash::sip::bench_u64 ... bench: 54 ns/iter (+/- 96) = 148 MB/s
```
After:
```
test hash::sip::bench_bytes_4 ... bench: 41 ns/iter (+/- 3) = 97 MB/s
test hash::sip::bench_bytes_7 ... bench: 48 ns/iter (+/- 0) = 145 MB/s
test hash::sip::bench_bytes_8 ... bench: 35 ns/iter (+/- 1) = 228 MB/s
test hash::sip::bench_bytes_a_16 ... bench: 45 ns/iter (+/- 1) = 355 MB/s
test hash::sip::bench_bytes_b_32 ... bench: 60 ns/iter (+/- 0) = 533 MB/s
test hash::sip::bench_bytes_c_128 ... bench: 161 ns/iter (+/- 5) = 795 MB/s
test hash::sip::bench_long_str ... bench: 514 ns/iter (+/- 5)
test hash::sip::bench_str_of_8_bytes ... bench: 44 ns/iter (+/- 0)
test hash::sip::bench_str_over_8_bytes ... bench: 51 ns/iter (+/- 0)
test hash::sip::bench_str_under_8_bytes ... bench: 52 ns/iter (+/- 6)
test hash::sip::bench_u32 ... bench: 40 ns/iter (+/- 2) = 200 MB/s
test hash::sip::bench_u32_keyed ... bench: 39 ns/iter (+/- 1) = 205 MB/s
test hash::sip::bench_u64 ... bench: 36 ns/iter (+/- 1) = 222 MB/s
```
The common pattern `iter::repeat(elt).take(n).collect::<Vec<_>>()` is
exactly equivalent to `vec![elt; n]`, do this replacement in the whole
tree.
(Actually, vec![] is smart enough to only call clone n - 1 times, while
the former solution would call clone n times, and this fact is
virtually irrelevant in practice.)
I added it because it was easy (same a `char::to_lowercase`,
just a different table), but it doesn’t make sense to have this
in std but not str::to_titlecase, which would require
https://github.com/unicode-rs/unicode-segmentation
At some point in the future this feature will be available
(both on char and str) in a crates.io crate.
As it says in the title. I've added an `expect` method to `Result` that allows printing both an error message (e.g. what operation was attempted), and the error value. This is separate from the `unwrap` and `ok().expect("message")` behaviours.
