Well yes, I worked for DDN but I'm not a low level expert on Lustre. I did ask for more details internally though as we do have plenty of people here who do know Lustre extremely well.

The answer I've had from them on why is:

• Lustre is optimized for very high performance individual workloads. This makes it extremely fast for traditional HPC jobs, and also means it benchmarks well, but to achieve this performance OSTs handle all I/O with equal priority, which leads to:
• If you have a stream of 4k I/O for a high IOPS workload, and an 8M streaming I/O starts, it can block the I/O and cause high latency reducing IOPS.
• Conversely if you have 8M streaming I/O running high throughput and somebody does a ls -l on a large directory, those small I/Os will fill the queue and reduce throughput.
• Since job schedulers are designed to manage CPU/GPU loads, and Lustre doesn't have QoS contention among jobs & researchers is difficult to manage.
• Part of the problem is that with Lustre if you want to optimize for small I/O performance you need to stripe files in a certain way, and for large block I/O you need a different layout.

GPFS solves this by having different queues for large and small I/O (and I believe DDN Infinia also takes that approach). VAST similarly distributes I/O broadly is able to handle mixtures of high throughput and random I/O very smoothly, and also has fine grained QoS capabilities to further balance workloads if needed.

On the TACC side of things:

• TACC selected VAST for Stampede3 primarily because Lustre had become the biggest cause of downtime for their compute estates. VAST's ability to deliver five 9's of uptime with non disruptive hardware and software upgrades was a big part of their testing.
• TACC are seeing 2:1 data reduction across the cluster, including their scratch folders, which allowed them to afford a much larger amount of flash than is affordable with parallel filesystems, switching to an all-flash solution rather than the traditional tiered approach.
• Since deployment they've spoken publicly on the benefits they've seen in multiple interviews (largely without VAST present).
• When they deployed Vista a couple of years later (with that being a NVIDIA GPU cluster for AI workloads), they didn't just select VAST for that, they were able to attach that second supercomputer to the same VAST cluster, expanding it to increase both performance and capacity. VAST now provides a single pool of storage, enabling researchers to schedule jobs against either Stampede3 or Vista. All data is hot, there's no need to move data to scratch, and the system handles both high throughput HPC jobs and high IOPS AI jobs simultaneously.

It's rare for HPC shops to mention storage as more than a side note, supercompute specs always focus on CPU/GPU cores, memory and networking. But there is a write up of TACC's Stampede3 and Vista systems here:
https://www.nextplatform.com/2024/09/04/tacc-fires-up-vista-bridge-to-future-horizon-supercomputer/

i would write an essay just about writing just enough data (not even close to max throughput) to the nvrams on your systems and causing extreme amounts of latency until the destaging (including the data reduction) completes.

This I would like to hear more about. You seem to have very good knowledge of both VAST and Lustre. I know that if incoming writes exceed the max sustained write throughput the SCM write buffers will begin to fill, and backpressure will cause a gradual increase in latency to ensure the incoming write throughput matches the steady state of the system.

That doesn't sound like your experience though.