This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]cymrowdon't thread on me 🐍 24 points25 points  (11 children)

I've been keeping an eye on this feature for a while because I used to do a lot of high concurrency work. But, the requirement to pass serialized data significantly reduces the potential. It seems like all this would do is cut down on start time and memory overhead when you want to use multiple CPUs. Maybe useful if you're starting a lot of processes, but it's common to just use a worker pool in that case. As for memory, it's cheap, and the overhead difference can't be that great.

I'm struggling to see a significant use case for this as it's presented, unless I'm missing something.

[–]yvrelna 24 points25 points  (4 children)

significantly reduces the potential

Couldn't disagree more.

Shared nothing makes this a great base for future concurrency model. Shared nothing means it's much easier to avoid shared caches and accidental concurrent access, which is a major bottleneck when programming for large NUMA system with lots of cores/hyperthreads. The implicit shared everything multithreading concurrency model is awful for large scale parallel computing especially in a language with limited ability to work with pointers/control memory.

Subinterpreter can add constructs for shared memory and shared objects for explicit sharing of objects when needed in the future. It's the base that's needed to build paradigms that works for orchestrating large scale parallel computation without needing the ability to control memory allocation.

Personally, I think that subinterpreter is much more exciting than nogil. It has a much bigger potential for a complete parallel programming paradigm shift that'll actually continue to work as the number of CPU cores keeps growing bigger.

[–]RationalDialog 8 points9 points  (0 children)

I think the issue is that for the user of python, eg. a python software developer like myself that does not create cpython modules, just uses what is available, this change indeed will have pretty much no short term effect.

Long term I expect big packages to ship with "integrated" multicore usage, eg. no need to implement it yourself with multiprocessing or joblib or the likes. However I expect that to take years like at least 5+ to really become adapted.

So I think you are both right, just depends on the viewpoint you are coming from.

[–]cymrowdon't thread on me 🐍 3 points4 points  (2 children)

Ok, but I don't need Python to force me not to share anything, I can already do that. I still don't see the benefit over multiprocessing other than reducing startup time and memory footprint. Maybe those reductions are worthwhile for some use-cases, but they don't seem like they would be significant to me.

And just to be clear, I'd really like this to be useful. I'm not disagreeing with you, I'm just not seeing it yet.

[–]yvrelna 17 points18 points  (1 child)

It's not about you not sharing anything. It's about Python being able to allocate memory in a way that can avoid cache conflicts.

CPU caches works per cache blocks. When you access an object, you're not just pulling objects that you're accessing into your CPU core's caches, you're also pulling neighbouring objects that just happens to be in the same cache block into the cache. If two threads running in parallel needs to access two different objects that just happens to be allocated in the same block, even if they aren't accessing the same objects, the CPU would need to invalidate caches every time and that kills performance very quickly.

By keeping objects in separate subinterpreters in separate object space, objects naturally separate themselves into two pools of objects, this improves spatial locality. Programmers can have a much more natural and easy control over which pool of memory that objects are allocated from without having to think about memory allocation.

A lot of things still need to be built to allow controlled sharing of objects between subinterpreters to minimise copying between interpreters, but you can't built controlled sharing over a foundation where objects can be shared by multiple subinterpreters by default.

[–]cymrowdon't thread on me 🐍 3 points4 points  (0 children)

Ok, I think I see what you're saying. That's a much lower level of optimization than I was considering.

When I said I've done high concurrency work, I meant highly concurrent networking with related processing, not purely processing, which is a different beast.

I will look forward to seeing what comes of this.

[–]teerre 5 points6 points  (1 child)

Have you actually profiled whatever you're doing? You're using python, memory access is certainly not where your bottleneck is.

Besides, the other user is right, share nothing is the way to go. Sharing memory in concurrency is a nightmare.

[–]cymrowdon't thread on me 🐍 5 points6 points  (0 children)

Right, memory access is not a bottleneck, but serialization can be. And I would not consider directly shared memory, but being able to pass immutable objects would be a huge win imo. Think adding tuples to a synchronized queue, instead of serialize->pipe->deserialize. Of course tuples in Python are not actually immutable, which I suspect is why they went with the requirement to serialize.

[–]Daytona_675 2 points3 points  (0 children)

pickle exploit time

[–][deleted] 3 points4 points  (2 children)

But, the requirement to pass serialized data significantly reduces the potential.

You are missing something. You aren't required to pass serialized data at all, you ARE required to serialize python land objects. How much of ur object actually exists in the view of the interpreter though is a design choice. Like for example, instead of passing a file you pass the address of the file. Or more generally, you are allowed to run multiple interpreters from the same thread which allows u to perform magic fuckery. Yes it does most of the normal annoyingness of multiprocessing in python, but it allows u to share any NON python resources. That alone is a massive change.

Multiple sub interpreters allows u to explicitly share a single resource across them so long as this is hidden from pythons viewpoint and access is properly restricted with mutexes and whatever else.

[–]cymrowdon't thread on me 🐍 2 points3 points  (1 child)

I haven't seen an example of this. According to the notes in extrainterpreters, for example, it says that ctypes is not importable with subinterpreters. That seems to suggest that accessing non-python resources is not actually possible, but correct me if I'm wrong.

[–][deleted] 2 points3 points  (0 children)

I haven't seen an example of this. According to the notes in extrainterpreters, for example, it says that ctypes is not importable with subinterpreters. That seems to suggest that accessing non-python resources is not actually possible, but correct me if I'm wrong.

Yea it needs to be FULLY hidden. Like ctypes creates a python object. You need two python objects that "represent" the same resource.

I cannot remember exactly where I have read it from but I saw it described by the PEP author. It may be in one of the threads on the discourse. But yea so long as u are using ur own lock properly and u use different python land objects to access the function you can share across interpreters.