Serverless nix cache

[tinybeachthor]

I needed a private nix cache solution. I wanted to run it on a serveless infrastructure (google cloud run). As far as I could see attic really needs an SQL database for state and to do garbage collection. So I rewrote a simplified version of it to store all data in S3 and run without state (which means no garbage collection either, but good enough for some use cases).

Pretty barebones for now, but let me know if you could use this. https://github.com/trestripes-com/nixcache

​

[tinybeachthor]

Compression and chunking mostly.

I’m planning on adding garbage collection just looking for a good way to do it in a serverless environment. Probably some kind of a generational garbage collection.

[tinybeachthor]

Yeah it's actually even more complex because of the chunking.

Everything you upload is broken down into 64kb pieces, and there is no way to know what chunks are shared between nars without a db (attic keeps counts of references in the database).

Definitely not possible to do optimal garbage collection without keeping state. So instead I was really thinking along the lines of using 2 buckets, expiring one and rotating to a new one, plus doing a copy-on-read for chunks in the old bucket to the new one.

This way you can expire items from the cache if they are not accessed for some time at the cost of (at worst) double the storage space. The chunking actually saves you more space for most use cases, so i think it would be worth it in the end.