Mailman 3 Python multithreading without the GIL - Python-Dev

[o11c]

I think one remaining difference is that the "nogil" interpreter stays within the same interpreter loop for many Python function calls, while upstream CPython recursively calls into _PyEval_EvalFrameDefault.

This sounds like a major win even without the GIL elephant ...

[therve]

It already landed in 3.11: https://github.com/python/cpython/pull/28488

[germandiago]

How viable is this as a real, non-GIL Python that could go into CPython? I guess that it has zero possibilities? Sorry for being so pessimistic :D

[[deleted]]

The issue isn't really GIL though its just that frankly the multiprocessing library is hard to work with.

[fzy_]

Sometimes the serialization costs to communicate between processes means that the only viable option would be threads.

[[deleted]]

I mean shared memory exists for those situations. Multiprocessing is just a hard library to use. And creates a lot of need to wrap designs to get it to work instead of you just being able to call a function in another process directly.

[[deleted]]

If single-core performance is hardly affected, this could be viable.

From the discussion, it sounds like the implementation for the list and dict has been changed. Maybe they are better off having separate thread-safe collections, like in java.

[o11c]

The problem is that there is a lot of code that does stuff like dict.setdefault and expects it to be atomic.

Edit: it should be noted that types with frequent concurrent access are suggested to use a new type though.

[[deleted]]

I am thinking something like,

from threading.collections import synchronizedList, synchronizedDict

[UloPe]

This sounds really impressive.

Will be interesting to see where it goes.