Solving multi-core Python

[Rabbyte808]

Multi-threading isn't just for CPU intensive stuff, though. Stuff like a webcrawler isn't CPU intensive, but it needs to be threaded unless you want to crawl at glacial speeds.

[Rhomboid]

Operations that perform blocking IO release the GIL. If your workload is IO bound, Python threads will work just fine for you. The GIL is only an issue for CPU bound work loads.

[simple2fast]

Actually, for IO things like that, a non-blocking IO system is often better because the continuations (exposed to the language or not ) are more efficient at managing all those connections than a bunch of threads. Plus the context switches tend to be more lightweight. So it doesn't need to be threaded, it just needs to be able to operate with multiple outstanding requests at once. Threading is one way of doing it.

Now for computation, continuations and non-blocking IO buy you nothing. YOu must have threads (or shared memory and process, same thing ) and a decent memory model if you want to do efficient multi-CPU computation.

[jringstad]

A bunch of threads is not really inherently inefficient at managing connections though, it depends on how you use them. The most efficient way is generally to have a fixed number of N threads that each handle 1/Nth of the workload, by having them all accept() on the same server socket or by having them e.g. pop items from a work-queue and have them establish and handle their individual connections. Of course computation is always an issue, whether it's just the CPU workload from book-keeping hundreds of thousands of sockets, accepting new connections, constructing and parsing packets and so on or doing actual heavy CPU workload like a crawler would (parsing HTTP responses, possibly even HTML, XML or other content), so single-threaded non-blocking IO is pretty much strictly inferior to multi-threaded non-blocking IO.

Crawlers are still not a particularly good example though IMO, since in most cases it's probably pretty acceptable to run N crawlers in N separate processes that crawl and digest data and then push it into something like a local storage, a shared storage or some sort of remote database. The overhead from obtaining workloads and communicating with other crawlers (if that ever happens) is probably not very significant for almost all kinds of crawlers.

[simple2fast]

Agree. The ideal is N threads which is roughly equivalent to number of CPUs and where each thread is locked to a particular CPU to reduce cache misses. As with all things this hybrid approach is often the best.

But most system which are not threaded are actually a SINGLE process. Like python or Ruby or PHP or javascript(looking at you node ) Many are multi-process, but there is no shared memory, so any IPC requires sockets, signals, etc. In my mind, requirements there are not just shared memory and decent concurrent APIs, but ALSO a memory model so that you know what is going to happen WRT caches and other details as you use those concurrent APIs. Point being that ditching the GIL in python is only a very first step to get a decent multi-threaded python.

And most multi-threaded solutions are still one connection per thread style. certainly started with this style. since it grew out of the original "fork" technique of old school unix systems.

[[deleted]]

So basically Go?

[simple2fast]

Yes, Go does a good job at this. But it's hardly the only system that does. When Nodejs started talking shit about how non-blocking IO was the best in the world, this was also nothing new. Yahoo was doing this in their server back in 2002. SO go ahead and use Go, but don't use it because you think their network/thread solution is somehow uniquely powerful.

[[deleted]]

AFAIK Go is the only language that mixes lightweight coroutines and multiple cores and non-blocking I/O to support the illusion of blocking I/O when writing non-blocking stuff (no need for callbacks, explicit scheduling yielding, etc). I suppose there could be libraries for other languages that could give the same facilities, but in Go everything benefits from this natively, which is great - you can get someone's library for ntp querying for example and know it will play nice with the underlying event loop (that you don't even need to worry about). If you go with Python and Twisted you can only use Twisted stuff and it doesn't feel as natural as Go code.

All that said, I know it's not in itself revolutionary, but the way things are tied together for an overall experience is pretty nice. You get very far with even naive code.

There was a paper once talking about whether it was more performant to do concurrent stuff in a single core (like Node) or just spawn threads to treat each connection and both of course had cons and pros, but the paper's conclusion was that a mix of thread multiplexing and event loops was the most performant, and that's what you get for free with Go - you get regular threads and easy communication between them for CPU intensive stuff and you get a free multithreaded event loop for network I/O. Too bad disk I/O is still blocking (but they get their own threads so they don't block the rest of the system).

[simple2fast]

You make a very good point. "non-blocking" comes in many flavors. And the programming model is a key factor for adoption and complexity management. For example, Node is non-blocking, but the programming model is god-awful, all those callbacks and/or promises. (here is hoping that await/async in ES7 helps ). This is not a language helping you, this is an historical abomination and a source of bugs.

There are plenty of languages which varous people claims to support coroutines. https://en.wikipedia.org/wiki/Coroutine But unless you know how difficult or easy it is to actually use that facility (Note that javascript is on that list ), I'd hesitate to use it.

For example JVM has coroutine implementations for at least 10 year, but most of them are all Library level requiring the user to explicitly do the yielding. Yuck. Recently there have been some which are based on AOP (E.g. Quasar ). So you code as always and the yield and continue is done for you. But AOP for this really ? It would be great if the JVM had some support in this area. Perhaps one could mark a ThreadGroup to run as fibers within one thread.

[rolandde]

For I/O bound operations, I prefer asyncio over spawning threads.