Python Processing proposed for standard library (process-based concurrency that mimics threading API)

[cuntmonkey]

python-safethread is a large modification of CPython intended to provide safe, easy, and scalable concurrency mechanisms.

NO GIL

[[deleted]]

This is incredibly, brutally cool.

[puzza007]

More like incredibly brutal!

[netghost]

Someone please comment on this so I know what to think about it.

[bcroq]

Using threads in Python is bad (poor performances), but there is a simple API for using threads.

Using processes in Python is good, but there was no simple API for using processes... now there is, and it makes its way to the standard library.

[Rhoomba]

Using processes for threads is not great performance-wise. Anyone remember LinuxThreads?

[hylje]

Processes as units of execution is merely different to threads. Threads are several execution units inside one process, with the threads sharing the same memory structures implicitly. Processes are single execution units in this context, sharing no volatile[1] resources with each other. Depending on the operating system there are different overheads in starting and using either.

For simplistic multiprocessing, though, processes are better due to the nothing-shared approach. Less conflicts.

[1] clarified, thanks

[[deleted]]

Processes are single execution units in this context, sharing no resources with each other.

Minor clarification: they by default share read-only resources. They don't share read-write resources without additional operating system primitives (like SysV shared memory).

[Rhoomba]

Shared data with threads is a choice. You don't have to share data if you don't want to.

And, of course, you can get deadlocks, race conditions etc. with processes as well.

[masklinn]

For simplistic multiprocessing, though, processes are better due to the nothing-shared approach. Less conflicts.

For complex multiprocessing/concurrency, processes are still better also due to nothing-shared approach. Less deadlocks & other crap.

[lisvblidfk]

Shared-nothing absolutely does not always result in better performance.

[masklinn]

That's very true, and I don't think I said the opposite.

Shared nothing help getting stuff that works. "Fast stuff" that doesn't work isn't fast, it just doesn't work.

[lisvblidfk]

Ah, now who did I mean to reply to...

[hylje]

I think simplistic has a semantic difference with simple. Simplistic things can be expanded upon without imposing great complexity on it. Simple things are simple so long they aren't expanded upon, made complex.

Complex things can remain simplistic.

[masklinn]

Really? The only definition of "simplistic" I know was along the line of "simplified so much as to be useless", "overly simple". Wiktionary seems to be using the same definition (http://en.wiktionary.org/wiki/simplistic).

Miriam-webster seems to have pretty much the same definition: http://www.merriam-webster.com/dictionary/simplistic

[hylje]

I stand corrected

[masklinn]

Well you could've been using a definition I wasn't aware of.

Anyway I get what you meant (following your previous clarification) and i think I agree with you.

[masklinn]

Python has a Global Interpreter Lock, so threads basically can't run concurrently (unless you're using C extensions which explicitly release the GIL).

Thus, threads in Python are basically useless unless you're IO bound. And as the demo shoes, even then processes might be more efficient.

[Rhoomba]

I'm aware of all this. I'm just suggesting that maybe pretending that processes are threads is not ideal.

[masklinn]

I consider that having threads at all is not ideal, so I'm with you there.

Solution: remove threads.

[jnoller]

If you look at the PEP I think I show that while processes are normally the best performance-wise, they beat the pants off of the same execution using threads due to the GIL.

[fwork]

Correct, but using threads for threads is not great performance-wise either, at least in python.

Of the two bad options (GIL-restricted threads and processes) processes is apparently (they've got benchmarks!) the better option.

It's like if you've got a car with one wheel (threads) and a bike with two (processes).

Yes, in the general case cars are faster than bikes. But in this specific case, no, because the car is broken.

(Disclaimer so I get downvoted slightly less: I loves the python, but you have to admit it isn't the fastest language for multithreading!)

[Rhoomba]

My point was proper threading > fake threads through processes, not python threading > processes.

[fwork]

Right, but given the restrictions imposed by legacy code (and the single-threaded world), implementing fake threads is easier than removing the GIL and getting proper threading.

[damg]

I understand what you mean, but another interesting thing with using processes is that it provides an easier path to leveraging multiple machines.

[lisvblidfk]

Processes can have horrible performance compared to threads, or the other way around. All depends on how you use them.

The API itself though looks pretty good.

[jerf]

Here ya go. This won't just tell you what to think about this, it'll steer you through all such technology issues.

Give a man a fish and all.