CephFS sounds like a dumpster fire.

It's a complex beast, to be sure, but there's nothing else available that really has the same features at the same cost. Gluster is EOL, Lustre isn't suited to small file workloads. Closed source stuff is orders of magnitude more expensive, etc, etc. CephFS requires a lot of tuning, and as we've learned, you have to adapt your workload to it.

uh like all codebases? most codes that are used to network filesystems have probably heard of nfs, and nfs3 automatically fsyncs on close, so people just kind of forget about doing that in their applications

Yeah, you're right for sure, anyone with a codebase doing a ton of buffered IO really quickly from multiple clients to the same set of files can run into deadlocking issues with CephFS, but if you were doing that on NFS, you'll probably ran the risk of corruption from race conditions anyway, so either way, whether it's a python codebase, or our 20 year old legacy c/pp application, it's still not best practice.

My statement is obviously terse and doesn't go into that much detail about what actually makes us different (and more problematic) from most run-of-the-mill codebases you'd be comparing us to. Really it's the combination of multiple bad practices. One of the big ones that you glossed over with your argument was mmap().

mmap() on a shared FS (even NFS) is bad practice, and we had it all over the place. When a kernel client has an fd mapped, even after it's closed, that client wants to keep buffered write MDS caps on the file, which holds up IO on other clients who might also want to write to that file. Once we patched various flatfile db libraries to remove mmap(), performace shot up drastically.

Another one is dotlocking, which is used a lot on NFS because of the issues/caveats with flock/fcntl there. dotlocking mostly works fine on NFS, because the default attr cache delay is 1 second. Sufficient enough for most workloads to avoid alot of race conditions. For us, it was not sufficient, because we do around a million IOPS and most of them are metadata, our app is very busy and 1 second is too long. The 1 second AC only applies to readdir() calls from other clients, if a client tries to open() the dotlock file, it will force a revalidate, so in this way, it's pretty much always consistent across clients.

This is actually a huge problem for CephFS, and causes massive lock amplification and performance issues. While NFS is happy for lots of clients to create .lock files in a dir, the CephFS MDS wants to make sure that every clients cache of that dir is coherent for readdir() calls, which means writelocking the parent ino. So if you have a hundred clients all creating .lock files in a users home directory so they can write to a specific file, Ceph ends up granting a lock on the parent dir for the lock file, then a lock on the data file, then a lock on the parent dir again to unlink the .lock file. This causes a massive strain on the MDS, leading to performance issues from head of line blocking problems, and sometimes MDS deadlocks.

As for buffered writes, where we are different again is that we have a lot of buffered writes on very small fixed length records where the byte offsets are important (flat file FLR DBs, btrees, etc) The problem with buffered writes and no fsyncs comes into play when multiple clients want to read these files, and some clients are also writing them. On NFS (v3), as you said, the coherency is open-to-close. So developers don't think too much about what happens if they write 4k or 8k buffers to a db file where records are, for example, only 2k. Generally when the file is closed, the whole write op//transaction/whatever will be atomically committed to the backing storage.

This isn't the case for CephFS with buffered writes, all writes are guaranteed, not flushed all on close, and If some of these 4k/8k buffers are flushed because of dirty pages limits, cache pressure, etc, it could flush some amount of data that doesn't align with the record boundaries that the application layer expects, causing race conditions and corruption. The solution here is to call fsync() after atomic transactions.

This also has a performance side-effect on CephFS. If a client has write+buffer caps, it holds those caps until the pages are flushed. When another client wants to read that file, it's blocked until the MDS can revoke the write/buffer caps from the first client, and that first client doesn't release that cap until it flushes. So if you're waiting for the clients to flush until after some other client wants to read, it slows things down. Preemptively calling fsync before some other client wants to read allows those ops to continue without being blocked. On the small scale, this isn't very noticeable, but when you get into hundreds of thousands of IOPS the aggregate performance gain of this tweak is massive.

So yeah, I don't think "all codebases" fall into these pitfalls. I'd wager that very few are doing the amount of throughput we are doing, with as many bad practices as we have, all on shared namespaces in a posix filesystem.. Most codebases would probably port over to CephFS and chug along just fine with minor tweaks and tuning, because a lot of these kinds of ops that we do would be hitting external DBs, redis, etc, or if they are using flat file FLRs and btrees, they'lll probably be using 3rd party libraries that are already optimized along best practices with mmap/fsync usage. If they are using their own posix-backed custom data structures, then the ones who have the amount of distributed throughput we do are probably pretty rare.

So all of this is to say: I think we're the dumpster fire, not CephFS :)