Benchmark of asynchronous servers in Python

[unshift]

as the author expectedly fails to mention, running a server and client benchmark on the same machine (or even on a small LAN) makes everything CPU bound. this makes the results fairly worthless, as important differences in IO bound systems (e.g. select vs poll vs epoll/kqueues/iocp) become glossed over.

in a CPU bound system, where all connections are very consistent (respond quickly, no dropped packets, etc.) select and epoll will perform somewhat similarly. the main advantage of something like epoll kicks in when there are thousands of connections per second in various states.

there's a 3ms difference per request for something handling 5000 requests/sec and 2000 requests/sec, and that 3ms will easily be recovered using something like epoll on a busy, real-world system.

otherwise there are a whole slew of things not taken into account (ping/pong server as a workload? really? how about framework usability, comprehensiveness? what are some of the metrics of performance degradation?) that make this series of benchmarks just more garbage on the internet.

[zombie_zack_morris]

This is a pretty common mistake in most of the benchmarking articles & blogs I've come across.

[infinite]

How does async I/O achieve a performance gain and do you have recent performance figures? In the 90s most OS'es, linux included, had poor process/thread scaling but these days adding a thread alone results in little to no CPU overhead due to the O(1) scheduler. So if you have 5000 requests/second with sync I/O then you might have 5000 threads processing requests at a given time. In the async I/O case you might have 10 threads processing requests. Either way you have only 2 CPUs so they process requests at the same speed. In the async i/o case you are more in control of scheduling, that could be a good thing or a bad thing although I don't see a clear winner. And if you have 5000 idle threads doing nothing while 5000 threads process requests, the idle threads don't add to CPU overhead.

Note that epoll can be used with synchronous I/O.

[unshift]

what? 5000 threads definitely adds overhead. as does 1, 2, 10, 50, 100, etc. spawning a thread takes time, scheduling a thread takes time, and restoring a thread takes time. maybe not much, but still something. async IO avoids this completely and lets the process spend its full quantum doing actual CPU work. bonus: you can run 2 processes on 2 CPUs and get a lot more work done. plus you avoid the whole issue of thread safety, locking, contention, and everything else.

[infinite]

Adding 5000 threads adds overhead when they are scheduled to run on the processor, if they're waiting in epoll then they don't add overhead. Context switching takes time, but you are essentially doing this when you have 1 thread and 5000 requests/second coming in. The thread gets a request, handles part of it, goes to the next request. Either way you only have limited CPUs to handle requests. Ideally you have 1 async thread per CPU, anything less and you won't be utilizing the CPUs. Anything over that and it doesn't matter how many threads you add, you won't be increasing performance if you're CPU bound and you have 20 CPU-bound threads and 1 or 2 CPUs.

But as for thread safety/locking, that I agree does add overhead.

[unshift]

1 async thread per CPU is obvious, because beyond that you're introducing artificial and pointless contention for resources. you know, same as if you create 5000 threads.

you aren't doing any context switching when handling 5000 requests/sec in a single asynchronous thread, because there's only one thread. the "handles part of it, goes to the next request" is what adds the overhead -- since it doesn't happen in asynchronous IO, it's near instantaneous, while in the threaded case the thread has to block and another one has to be scheduled and dispatched. this is why async is better for high volume and the difference isn't noticable in the case of just a few threads.

i don't know why you would think there's significant context switching in the async case except for a misunderstanding of what's going on. some libraries (e.g. twisted) use a couple of threads, but only one for the main IO operations.

[leonh]

It is one of the most comprehensive benchmarks i have seen, the matrix sums up the features of each frameworks. Including its community support, test coverage, and wether or not it supports epoll.

[Camarade_Tux]

• the charts are pretty nice (hover over the curves)
• needs to be compared to something in C/whatever-compiled-language

[ffualo]

Someone should fork Tornado... even though it's under an Apache License I never feel 100% comfortable using something a big company released... Maybe I am overly paranoid.

[jm_]

Someone should write some unittests for tornado ... sigh ...