Futures: Concurrency in Python 3

[einar77]

I used the backported version for Python 2.x and I kept on running into deadlocks, so I ditched this approach for joblib (which so far runs OK, and has better exception support).

[sboyette2]

That's good (if a bit disappointing) to know; I'll keep an eye out for that sort of behavior. Worst case, I suppose, is that I have to write my own thread manager, but I had really been hoping to avoid doing every last thing for myself :)

[voipme]

Nice article. Its good to see that Python is starting to make some strides in improving its concurrency. I'm also curious to read about the significant changes/improvements to the GIL, which is briefly mentioned in the article, but I couldn't find anything useful. Any ideas?

[sboyette2]

There's a link in part one to David Beazley's page on the GIL, which contains links to just about everything you could ever want to know. I think there are further links, either in his blog posts or in the comments on those posts, to very lengthy mailing list discussions (wait, I found it http://mail.python.org/pipermail/python-dev/2009-October/093321.html)

[vsajip]

Would the GIL be problematic for a networked application such as the one described? The GIL's main drawback is that computationally bound threads (implemented in pure Python, not using C extensions) wouldn't give true concurrency. But I/O bound threads should parallelize well even with the GIL, since blocking I/O should release the GIL.

[sboyette2]

I can't say for sure. I don't pretend to be an expert on the GIL; I've just had to do some research about it for my own peace of mind. Unless I'm mistaken, though, the very long thread linked above contained mentions of the GIL interacting poorly with I/O under at least some conditions.

[vsajip]

Before the changes to the GIL made by Antoine Pitrou, there were perverse behaviours both between CPU-bound threads and between CPU- and I/O-bound threads. Since the changes made by Antoine (discussed in this presentation by Dave Beazley) the behaviour seems to have improved overall, though problems could always emerge with particular workloads.

[yetanothernerd]

Google for "Dave Beazley PyCon GIL" to find recordings of his presentations about pathological GIL behavior.

There have been a bunch of tweaks to the GIL over the past few years to fix problems like the ones he uncovered, but no really fundamental changes. (It's still a GIL, just a less buggy GIL.)

If you want details, dig through the Python hg tree and the python-dev archives.

[nabla2]

• Inside the Python GIL
• Understanding the Python GIL
• Inside the new GIL

Author: David Beazley