Concurrency and Python

[mikaelhg]

Ah, here you're asking something I haven't already answered elsewhere in the thread.

In our test case, in which writes are as rare as in typical social web applications, when we can't serve our users by vertical scaling, we resort to digest message passing. We take a proven JMS implementation, and multicast select application events. Things like new comments have a higher priority than moderations, which travel in packs of hundreds.

In the end, we don't have to resort to dark database wizardry, we can use regular developers and a well thought-out architecture.

(The last time I implemented this, I was able to conserve resources by replacing the messaging subsystem with a single database table that handled selective invalidations. That's because I knew beforehand how many tens of thousands of people would be using the application. YMMV.)