14 Commits

Author SHA1 Message Date
bors
30e49a9ead Auto merge of #75272 - the8472:spec-copy, r=KodrAus
specialize io::copy to use copy_file_range, splice or sendfile

Fixes #74426.
Also covers #60689 but only as an optimization instead of an official API.

The specialization only covers std-owned structs so it should avoid the problems with #71091

Currently linux-only but it should be generalizable to other unix systems that have sendfile/sosplice and similar.

There is a bit of optimization potential around the syscall count. Right now it may end up doing more syscalls than the naive copy loop when doing short (<8KiB) copies between file descriptors.

The test case executes the following:

```
[pid 103776] statx(3, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_ALL|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=17, ...}) = 0
[pid 103776] write(4, "wxyz", 4)        = 4
[pid 103776] write(4, "iklmn", 5)       = 5
[pid 103776] copy_file_range(3, NULL, 4, NULL, 5, 0) = 5

```

0-1 `stat` calls to identify the source file type. 0 if the type can be inferred from the struct from which the FD was extracted
𝖬 `write` to drain the `BufReader`/`BufWriter` wrappers. only happen when buffers are present. 𝖬 ≾ number of wrappers present. If there is a write buffer it may absorb the read buffer contents first so only result in a single write. Vectored writes would also be an option but that would require more invasive changes to `BufWriter`.
𝖭 `copy_file_range`/`splice`/`sendfile` until file size, EOF or the byte limit from `Take` is reached. This should generally be *much* more efficient than the read-write loop and also have other benefits such as DMA offload or extent sharing.

## Benchmarks

```

OLD

test io::tests::bench_file_to_file_copy         ... bench:      21,002 ns/iter (+/- 750) = 6240 MB/s    [ext4]
test io::tests::bench_file_to_file_copy         ... bench:      35,704 ns/iter (+/- 1,108) = 3671 MB/s  [btrfs]
test io::tests::bench_file_to_socket_copy       ... bench:      57,002 ns/iter (+/- 4,205) = 2299 MB/s
test io::tests::bench_socket_pipe_socket_copy   ... bench:     142,640 ns/iter (+/- 77,851) = 918 MB/s

NEW

test io::tests::bench_file_to_file_copy         ... bench:      14,745 ns/iter (+/- 519) = 8889 MB/s    [ext4]
test io::tests::bench_file_to_file_copy         ... bench:       6,128 ns/iter (+/- 227) = 21389 MB/s   [btrfs]
test io::tests::bench_file_to_socket_copy       ... bench:      13,767 ns/iter (+/- 3,767) = 9520 MB/s
test io::tests::bench_socket_pipe_socket_copy   ... bench:      26,471 ns/iter (+/- 6,412) = 4951 MB/s
```
2020-11-14 12:01:55 +00:00
The8472
16236470c1 specialize io::copy to use copy_file_range, splice or sendfile
Currently it only applies to linux systems. It can be extended to make use
of similar syscalls on other unix systems.
2020-11-13 19:45:27 +01:00
Benoît du Garreau
001dd7e6a5 Add tracking issue 2020-11-06 18:04:52 +01:00
Benoît du Garreau
ae059b532f Make some std::io functions const
Includes:
- io::Cursor::new
- io::Cursor::get_ref
- io::Cursor::position
- io::empty
- io::repeat
- io::sink
2020-11-06 17:48:26 +01:00
ecstatic-morse
65bdf79da3
Rollup merge of #76275 - FedericoPonzi:immutable-write-impl-73836, r=dtolnay
Implementation of Write for some immutable ref structs

Fixes  #73836
2020-09-21 20:40:44 -07:00
Federico Ponzi
88a29e630c
Updates stability attributes to the current nightly version 2020-09-21 08:52:59 +02:00
Federico Ponzi
28db5214d2
More implementations of Write for immutable refs
Fixes #73836
2020-09-03 09:36:05 +02:00
Ralf Jung
0af3bd01df Read: adjust a FIXME reference 2020-09-02 12:34:15 +02:00
bors
d9cd4a33f5 Auto merge of #76047 - Dylan-DPC:rename/maybe, r=RalfJung
rename get_{ref, mut} to assume_init_{ref,mut} in Maybeuninit

References #63568

Rework with comments addressed from #66174

Have replaced most of the occurrences I've found, hopefully didn't miss out anything

r? @RalfJung

(thanks @danielhenrymantilla for the initial work on this)
2020-09-01 05:41:22 +00:00
Lzu Tao
a4e926daee std: move "mod tests/benches" to separate files
Also doing fmt inplace as requested.
2020-08-31 02:56:59 +00:00
DPC
b3d7b7bdcb update fixmes 2020-08-30 14:43:52 +02:00
DPC
5e208efaa8 rename get_{ref, mut} to assume_init_{ref,mut} in Maybeuninit 2020-08-29 02:13:02 +02:00
Alexis Bourget
5d49c0e55a Move to intra doc links for std::io 2020-08-18 19:36:52 +02:00
mark
2c31b45ae8 mv std libs to library/ 2020-07-27 19:51:13 -05:00