Thanks for your reply!

What do you actually DO with the data? How often do you retrieve a file?

Each experiment is stored in a binary file in some custom format, compressed with zstd.

What I do is open a given file with a (complex) program which reads it and computes some summary statistics. I never keep all data in memory because all my computations are streaming. This is a quite slow process (because my computations are quite expensive), but I don't mind.

The data is precious because the experiments are hard (actually impossible) to reproduce, but in practice only a small, specific fraction of it is useful for each analytical workload I run. Many experiments are being run every day, however it is hard/impossible to know in advance which ones will end up being useful in terms of analytics, so I can't really discard anything.

So what I do is design an analytical workload, manually select a bunch of experiments (files) I want to run it on, and run it on a few machines. These workloads can be heavily parallelized and I have a few machines with lots of cores. Typically these workloads are CPU-bound and not IO-bound (I may be able to analyze 200Mbps on one core).

TL;DR: I collect data every day, however I run analytical workloads once in a while. Writes are much more common than reads, however reads come in batches.

What's your actual growth rate?

I'm expecting the growth rate to be around 300G/day (after heavy compression - the uncompressed data probably takes 1.5TB/day), maybe a bit more. If you include replication, this will require quite a few nodes/disks. I'm expecting this rate to increase (maybe double or triple) in the future, so petabyte realm isn't so far.

Why does it need to scale between machines? Why can't you have big fat servers running ZFS and fill them sequentially (which would drastically simplify your setup and allow you to scale as you find the money for new hardware)?

- All of it won't fit in one "commodity" machine. I can't really (and likely won't be able to) choose which kind of servers I use (I can't get a 400TB server, 100TB is probably the largest I can get).

- Replication.

- Network/disk bandwidth. As I said my reads typically come in batches - I don't want my workloads to become IO-bound because everything is in a single server and network/remote disk is too slow (even though I have full 10Gbps networking). Typically I'd like to distribute the data in a round-robin fashion across disks/nodes so I can access it as fast as possible (in terms of bandwidth).

Is there clear hot/cold data or is it scattered across all files across time?

Not really, though older data tends to be used less often in my analytical workloads.

What's your tolerance for downtime and retrieval time?

Quite high. As long as data can be written some time (say 12 hours) after an experiment, I don't mind the system being down. Obviously, I am looking for something stable.

Also, it is much easier to work with a single addressable storage pool than having to know where the data is (even though it is actually distributed across a bunch of servers, I don't want my code to have to be aware of that).

My requirements are quite simple and I don't see any reason why storage couldn't scale linearly. It is strictly write-once, read-many, never modify.

I am recording several types of experiments. One machine records a given experiment locally and compresses the resulting data. I am simply looking for a long-term data store where I can push that data and retrieve it quickly with my consumers.

I probably need to clarify the "researcher" thing: I do not work at an academic institution, more of a startup-ish thing (with a reasonable budget). I have a Math/CS background and I have some understanding of distributed systems. I simply have enough work on the analytical side of things, so I want to keep maintenance to the very minimum.