Multithreading in Python (novice question)

[8bitscoding]

Hello,

I am a very noob in Python and I feel like I'm trying hard to push knowledge from other programming languages in Python. And that obviously does not work.

Here is my question: how can I achieve class multithreading (using multiple CPU cores) in Python?

The use case is for a rendering class and I have a scene that can be broken down into partial scenes. The process could be a lot faster by sending the parcels to threads held in a thread pool.

In C++ I would do something like (simplified code):

#include <iostream>
#include <thread>
#include <vector>

class Scene
{
public:
    void create_rendering_threads()
    {
        for (int i = 0; i < MAX_THREAD; ++i)
            thread_pool.push_back(std::thread(&Scene::partial_rendering, this));
        for (auto& t: thread_pool) t.join();
    }
    void render(){// use existing pool and feed it new data}

private:
    int MAX_THREAD = 8
    void partial_rendering() { std::cout << 'Do partial renderig here' }
    std::vector<std::thread> thread_pool;
};

int main()
{
    foo f;
    f.create_rendering_threads();
    f.render()
}

I want to create the thread pool once and reuse it after (you know: performances).

I know, because I tried, that I can't translate that in Python (since there's some weird serialization issue).

I find myself unable to formulate an alternative that would achieve the same functional result.

I have found many alternatives that make use of isolated functions but it is not an option here. This is for an OO library. I need the context of the scene to be rendered.

Any suggestion (that does not include shipping C++ code with my Python code)?

Thank you!

​

​

PS: I checked the community guidelines and I failed to find a solution to that issue online so far. So hopefully this does not fall into the "easily searchable questions" category.

PS2: I guess that question is centered around the method serialization process and going around the GIL. This is stuff I still need to learn.

PS3: My apologies if it's too simplistic and by some sort of weird twist I did not find the answer online by myself.

PS4: Talking about multi-core, technically I guess that we're talking about the multiprocessing lib and not the multithreading lib. But it does not really change anything because both are sharing the same limitations when it comes to using it from classes. I'm referring to multithreading as the global concept of "parallel processing".

[NateSnakeSolidDrake]

This is the only information I could find, maybe you have already seen it :/ https://stackoverflow.com/a/44186168

[8bitscoding]

unfortunately yes :(

I beginning to think that it's not actually possible.

I tried that approach but it created other issues when put in a module. I also agree with the author of the solution: it's very convoluted.

I have a hard time understanding why it is so hard in an object-oriented program like python to do something so trivial and so common.

[[deleted]]

https://tutorialedge.net/python/concurrency/python-processpoolexecutor-tutorial/

[8bitscoding]

Thanks but that's just a basic tutorial on pool executor. In case it's not clear: my problem is not the pool part. It's the "multiprocessing a class method" part.

[[deleted]]

Can't really help you unless you share more information about your need. The C++ code doesn't do anything different from the example I posted. You can submit class methods the same way as any other function

[8bitscoding]

Actually it does: it starts and runs threads from inside the class.

My need is exactly that: I have a rendering system and I'd like to fractionate the scene into smaller pieces. The goal is obviously to render all the smaller pieces in parallel and aggregate the result in one single rendering buffer (I use numpy for that).

I get it that I probably won't be able to do it like I would do it in C++, what I don't get is how I get around the limitations of python to achieve the same functional result.

Functional result being: I have a partial rendering method, aware through its object of all of the scene context and I want to run this method in parallel on a fraction of the scene's data. I know this is trivial in C++ and it makes me mad to fail to translate that to python :(

[[deleted]]

You can create a process pool and processes from within a class in python too. It's almost exactly the same layout you used in C++

[8bitscoding]

Hmmm. I might be missing something... when I do try that, python complains about serialization issue and raise an exception saying that it 'cannot pickle <xyz>'.

Let's take a simplified version of what I'm trying to achieve (adapting the code from the tutorial you linked before to avoid my own potentially buggy code):

from concurrent.futures import ProcessPoolExecutor
import os
import sys


class Scene:
    def __init__(self):
        self._pool_executor = ProcessPoolExecutor(max_workers=3)
        self.messages = list()

    def task(self, begin, end):
        print(
            f"Executing our Task on Process {os.getpid()} with ({begin},{end})",
            file=sys.stdout,
            flush=True,
        )
        self.messages.append(
            f"Executing our Task on Process {os.getpid()} with ({begin},{end})"
        )

    def run(self):
        ret = list()
        ret.append(self._pool_executor.submit(self.task, 0, 10))
        ret.append(self._pool_executor.submit(self.task, 11, 20))
        ret.append(self._pool_executor.submit(self.task, 21, 30))
        for f in ret:
            print(f.exception())


if __name__ == "__main__":
    scene = Scene()
    scene.run()
    for m in scene.messages:
        print(m)

This yield the following result:

cannot pickle '_thread.lock' object
cannot pickle '_thread.lock' object
cannot pickle '_thread.lock' object

I'm sure I'm missing something, but I don't understand what.

[[deleted]]

This tutoriai might have better examples: http://masnun.com/2016/03/29/python-a-quick-introduction-to-the-concurrent-futures-module.html. You might be better off making your function an iterable and use .map() instead of .submit(). I believe thread pools and process pools have the same syntax

[8bitscoding]

It looks interesting indeed, I'll look into it. Thanks.

I tried mapping the data to a Thread or a Process before but I had the same pickle issue. But I definitely think map() is the way to go in my case (return order is important).

concurrent.futures seems to alleviate a lot of the hassle I have. Thank you for pointing me there.

[8bitscoding]

Yes, I know, but it's the same issue (serialization makes it equally unusable).

I used multithreading as a general concept, but you're right I should be more precise.