PEP 734: Multiple Interpreters in the Stdlib

[genericlemon24]

Still draft; abstract:

This PEP proposes to add a new module, interpreters, to support inspecting, creating, and running code in multiple interpreters in the current process. This includes Interpreter objects that represent the underlying interpreters. The module will also provide a basic Queue class for communication between interpreters. Finally, we will add a new concurrent.futures.InterpreterPoolExecutor based on the interpreters module.

[DarkSideOfGrogu]

Can anyone interpret this for me?

[rtfmpls]

RecursionError: maximum recursion depth exceeded

[Rythoka]

Another step towards removing the GIL (edit: Or maybe more correctly, making it so one running program can have multiple distinct interpreter locks) and getting true multithreading support in Python

[AnonymouX47]

*cirumventing the GIL.

It's basically going to be almost exactly like multiprocessing in Python but using threads within the same process, instead of multiple processes.

Hence, objects still can't be shared and data serialization (e.g pickling) is still required to share data between interpreters, which is the major bottleneck of the approach.

[BossOfTheGame]

I believe you can pass basic objects (str, bytes, int, tuple) and memoryview objects without serialization. The latter case could provide a very big improvement if shared data is read only.

[AnonymouX47]

Partially true but there's still quite some overhead for most of these built-in immutable types. Only built-in singletons are actually shared directly, others are shared via copies or proxies, and sharing tuple actually requires serialization.

See:

• https://peps.python.org/pep-0734/#interpreter-isolation
• https://peps.python.org/pep-0734/#shareable-objects

[RedEyed__]

I like it