proposal: archive/tar: support zero-copy reading/writing

[hanwen]

Proposal Details

the container ecosystem (podman,docker) spends its days creating and consuming huge .tar files. There is potential for significant speed-up here by having the tar package use zero-copy file transport.

The change is straightforward, but involves an API change, so opening a proposal.

with the following change, tarring up a 2G file from tmpfs to tmpfs goes from 2.0s to 1.3s

diff -u /home/hanwen/vc/go/src/archive/tar/writer.go hacktar/writer.go
--- /home/hanwen/vc/go/src/archive/tar/writer.go	2024-08-22 14:56:29.586690369 +0200
+++ hacktar/writer.go	2024-12-12 15:01:22.150045055 +0100
@@ -9,6 +9,7 @@
 	"fmt"
 	"io"
 	"io/fs"
+	"log"
 	"path"
 	"slices"
 	"strings"
@@ -491,7 +492,7 @@
 //
 // TODO(dsnet): Re-export this when adding sparse file support.
 // See https://golang.org/issue/22735
-func (tw *Writer) readFrom(r io.Reader) (int64, error) {
+func (tw *Writer) ReadFrom(r io.Reader) (int64, error) {
 	if tw.err != nil {
 		return 0, tw.err
 	}
@@ -550,6 +551,16 @@
 }
 
 func (fw *regFileWriter) ReadFrom(r io.Reader) (int64, error) {
+	log.Println("hanwen")
+	if _, ok := fw.w.(io.ReaderFrom); ok {
+		n, err := io.Copy(fw.w, r)
+		if n > fw.nb {
+			return n, fmt.Errorf("read %d bytes, beyond max %d", n, fw.nb)
+		}
+		fw.nb -= n
+		return n, err
+	}
+
 	return io.Copy(struct{ io.Writer }{fw}, r)
 }
 

[hanwen]

A similar optimization exists for the reading side, of course.

[ianlancetaylor]

Just to spell it out, I believe that the API change here is to define a new method on archive/tar.Writer:

// ReadFrom implements [io.ReaderFrom].
func (tw *Writer) readFrom(r io.Reader) (int64, error)

Note that I think you could get a similar effect without the API change by writing

	if tw, ok := fw.w.(*Writer); ok {
		return tw.readFrom(r)
	}

CC @dsnet

[hanwen]

your suggestion certainly improves regFileWriter.ReadFrom, but nobody calls that unless Writer.ReadFrom is exported. Am I missing something?

[hanwen]

for the reader, this works.

 // TODO(dsnet): Re-export this when adding sparse file support.
 // See https://golang.org/issue/22735
-func (tr *Reader) writeTo(w io.Writer) (int64, error) {
+func (tr *Reader) WriteTo(w io.Writer) (int64, error) {
 	if tr.err != nil {
 		return 0, tr.err
 	}
@@ -688,6 +688,12 @@
 }
 
 func (fr *regFileReader) WriteTo(w io.Writer) (int64, error) {
+	_, ok1 := fr.r.(io.WriterTo)
+	wrf, ok2 := w.(io.ReaderFrom)
+	if ok1 && ok2 {
+		return wrf.ReadFrom(&io.LimitedReader{R: fr.r, N: fr.nb})
+	}
 	return io.Copy(w, struct{ io.Reader }{fr})
 }
 

[ianlancetaylor]

It's probably me that was missing something.

[mvdan]

cc @dsnet given the TODO above

[Jorropo]

Do we want to include logic to pad out the tar to align content files to the destination's blocksize if it's an *os.File in the writer ?
Performance improvements go from single digit x improvement to multiple thousands through reflink at the cost of making the exact bytes of the tar dependent on the output.

Tar natively pad to 512 :'(

go/src/archive/tar/format.go

Line 143 in e39e965

blockSize = 512 // Size of each block in a tar stream

[hanwen]

@Jorropo - Fascinating insight, thanks! I can confirm that on btrfs, if I set the blockSize to 4096, I can write a 2G tar file in 0.08s which is amazing.

Unfortunately, it appears that the block size is not variable in the tar format, so this needs to be done in a different way. Fortunately, one could simply add as many empty files to pad out the tar file to 4096 or whatever the destination block size is. This can be done without changing the tar package at all.

[Jorropo]

I am not suggesting we change that field, 512 is hardcoded part of the tar format.
If we want to do this we might be able to figure out a way to inject no-op fields parsers will ignore in order to bump in the correct 512 bucket to be 4KiB (or whatever) alligned.

[hanwen-flow]

If we want to do this

having the bytes generated depend on a non-obvious property of the destination sounds confusing to me; I think we wouldn't want to do this. Regardless, it should go into its own proposal. Let's not side-track this discussion.

[hanwen]

A change with some numbers at https://go-review.googlesource.com/c/go/+/642736. For tmpfs and ext4, it yields a 10-20% speed improvement.

I have trouble measuring consistent results with BTRFS on Fedora 41. A standalone program can seemingly copy around large files no time at all (speeds of ~1000 Gb/s), but these aren't reflected in the benchmarks. Despite this, strace shows copy_file_range being executed in the same way between the standalone program and the benchmark.

[hanwen]

it looks like btrfs needs to have fsync called on the file, for it to be eligible for metadata only copies. Adding src.Sync() to the writeLargeFile function yields

2025/01/15 17:39:27 wrote input0 (1310720000 byte) in 1.64345491s: 0.742766 gb/s
2025/01/15 17:39:27 copy input0 -> output0: 1310720000 bytes in 483.089µs: 2526.870049 gb/sec
2025/01/15 17:39:29 wrote input0 (1310720000 byte) in 1.873767098s: 0.651470 gb/s
2025/01/15 17:39:29 copy input0 -> output0: 1310720000 bytes in 310.081µs: 3936.723388 gb/sec
2025/01/15 17:39:30 copy input0 -> output-2-0: 1310720000 bytes in 137.397µs: 8884.496204 gb/sec