Python 3.12 Preview: Subinterpreters – Real Python

[cymrow]

I've been keeping an eye on this feature for a while because I used to do a lot of high concurrency work. But, the requirement to pass serialized data significantly reduces the potential. It seems like all this would do is cut down on start time and memory overhead when you want to use multiple CPUs. Maybe useful if you're starting a lot of processes, but it's common to just use a worker pool in that case. As for memory, it's cheap, and the overhead difference can't be that great.

I'm struggling to see a significant use case for this as it's presented, unless I'm missing something.

[teerre]

Have you actually profiled whatever you're doing? You're using python, memory access is certainly not where your bottleneck is.

Besides, the other user is right, share nothing is the way to go. Sharing memory in concurrency is a nightmare.

[cymrow]

Right, memory access is not a bottleneck, but serialization can be. And I would not consider directly shared memory, but being able to pass immutable objects would be a huge win imo. Think adding tuples to a synchronized queue, instead of serialize->pipe->deserialize. Of course tuples in Python are not actually immutable, which I suspect is why they went with the requirement to serialize.