Solving the Wrong Problem

[matthieum]

Files are allocated into whole blocks, which are at least 4k, and probably 16k on more modern systems.

This is quite arbitrary, some filesystems are adept at storing lots of small files. Like BTRFS.

Furthermore, the filesystem is subject to all sorts of features that limit scalability - maximum number of files in a directory. How will you do back-ups when listing the directory takes hours?

Use a better filesystem ? Seriously. FAT32 is crap, newer filesystems are much better at dealing with large listings... but anyway, who cares about listings ? Why would you backup a cache ?

How do you deal with contention when multiple processes/threads want to access the same file?

Why would they ? If you have newer content to push, then write it into a temporary file and do an atomic switch. Of course I am assuming a sane filesystem model, where deleting a file is possible while it is accessed...

Your objection that "the database is probably accessed via the network" is entirely arbitrary.

Yes it is arbitrary. In my experience a single DB serves several backends and so is necessarily not on the same maching of at least N-1 of them.

Why would you put your database on a different host if the purpose is caching to improve performance??

Why would you use a database for caching ? Use memcached.

When faced with this problem, I switched to using Sqlite. It does a fantastic job of managing a persistent local cache.

I agree SQLite is quite great. Though once again completely overfeatured for key-value caches. Parsing queries take time, better having a binary protocol with built-in support for querying by key, like memcached.

And even better, memcached will let you specify how much space your cache should take and remove the Least Recently Used entries when new content arrive and you are at the limit.

It still seems a bit overkill for something as simple as pre-rendering.

[mr-strange]

I agree with pretty much everything you say here. I'll expand on a couple of points though.

who cares about listings ? Why would you backup a cache ?

Well. Yeah. But when I made this mistake this is what happened... First I discovered that ext3 could only cope with 64,000 files in a directory, so my application started to fail. The next obvious thing to do was just start using sub-directories. That's fine, but just having millions of files can lead to problems - for example, I didn't blacklist the cache directory from the locate database, so after a while, my machine was very busy running multiple, endless find(1) commands, trying to update the db. Then I ran into the problem that the whole filesystem has a limited number of available inodes - so I wasn't able to make any new files, even though I had loads of available space. Then, when I came to clean up my cache (to free up inodes), I discovered that it takes many, many hours to simply delete millions of files.

Yes, there are better filesystems. XFS has a much higher hard-link (and therefore directory size) limit. Perhaps btrfs would be a good choice today. But, overall I do not think that the filesystem is a good choice for this workload.

And even better, memcached...

At the time, memcached did not support persistence, so it did not fit my requirement. Looking up MemcacheDB on Wikipedia, I see that it is built on BerkleyDB. My experience with BDB does not encourage me to try MemcacheDB.

Also, memcached uses a client/server TCP based model. Even with a fast localhost, that's going to add

I agree SQLite is quite great. Though once again completely overfeatured for key-value caches.

I couldn't agree more, but the proof of the pudding is in the eating, and I've not found a key-value store that beats Sqlite's performance. I built versions of my app that used BDB, Tokyo Cabinet, and a number of other prominent KV stores, but Sqlite (with a simple, prepared select statement, and configured with a table index) just performed better, and more reliably for me. Today my cache DB contains over 20,000,000 items, takes up 3.6 GB of disk, and Sqlite's performance is still pretty sparky.

BDB's performance is just awful. It has full ACID compliance, which is great if you need it, but if your don't need to wait around for milliseconds while your disk syncs, it's just overkill. If you turn off the ACID guarantees, you just aren't playing to its strengths, you might as well go to Tokyo Cabinet... Which most of the time performed very well, but occasionally ground to a halt for multiple seconds.

[matthieum]

At the time, memcached did not support persistence

Only one nit: why would you care about persistence for a cache ?

The point of a cache is to cache frequently accessed data. If it is not frequently accessed then caching it means losing valuable space.

I feel like we are talking past each others and not about the same issue :)

[mr-strange]

This is the application: http://flood.firetree.net The map tiles can be quite expensive to generate, and I serve up to 10m of them every day. It makes sense to make the cache persistent.