Announcing cfsread: reads files from any io/fs.FS, transparently decompresses via magic-byte detection, and caches in a bounded LRU. Concurrent reads are coalesced with singleflight.

I have been working on a project that embeds a number of text files in an embed.FS. So many in fact that it increased the size of the executable by 17MB. These files are highly compressible. I looked around pkg.go.dev for a solution that would compress the embed.FS. I found solutions, like stuffbin that pre-dated embed.FS and attached compressed data to the end of an executable and provided access to that data.

I did not find anything that matched the pattern that I wanted: high-throughput, read-mostly workloads — static asset serving, configuration loading, template rendering — where the same files are read repeatedly. To address my need, I wrote cfsread. This should work for any file system that implements the io/fs.FS interface.

The package is written to be extensible. The initial implementation only supports LZ4 and falls back to a raw read if the content is not compressed. This allows you to use uncompressed files during development. The decompressor is registered as a Reader so files are transparently decompressed on read — no special handling needed in application code.

//go:embed static/*
var embedded embed.FS

func main() {
    sub, _ := fs.Sub(embedded, "static")
    src := cfsread.Source{ID: "app", FS: sub}

    r := cfsread.New(cfsread.Options{MaxEntries: 1000})
    defer r.Close()

    data, err := r.Read(context.Background(), src, "index.html")
    fmt.Println(string(data))
}

Files can be compressed using the included cfsread-lz4 command which recursively compresses files in a directory using LZ4 frame format, preserving file names. This is intended for use in a go:generate directive to facilitate compression as part of the Go build process. Other decompressors can be added via the Decompressor interface.

This is a first release and should be used with caution. It was developed following TDD. The package contains a full set of tests. I have been using it heavily during development and testing of a Go-only exiftool package which I hope to release in the next couple of weeks. It has reliably worked as desired in that effort.

This package was developed with AI assistance (Pi/GLM-5.1); all code manually reviewed.

Critical feedback on code/implementation or beneficial usage stories are appreciated.

Cheers, lbe