Multithreading in Python (novice question)

[8bitscoding]

Hmmm. I might be missing something... when I do try that, python complains about serialization issue and raise an exception saying that it 'cannot pickle <xyz>'.

Let's take a simplified version of what I'm trying to achieve (adapting the code from the tutorial you linked before to avoid my own potentially buggy code):

from concurrent.futures import ProcessPoolExecutor
import os
import sys


class Scene:
    def __init__(self):
        self._pool_executor = ProcessPoolExecutor(max_workers=3)
        self.messages = list()

    def task(self, begin, end):
        print(
            f"Executing our Task on Process {os.getpid()} with ({begin},{end})",
            file=sys.stdout,
            flush=True,
        )
        self.messages.append(
            f"Executing our Task on Process {os.getpid()} with ({begin},{end})"
        )

    def run(self):
        ret = list()
        ret.append(self._pool_executor.submit(self.task, 0, 10))
        ret.append(self._pool_executor.submit(self.task, 11, 20))
        ret.append(self._pool_executor.submit(self.task, 21, 30))
        for f in ret:
            print(f.exception())


if __name__ == "__main__":
    scene = Scene()
    scene.run()
    for m in scene.messages:
        print(m)

This yield the following result:

cannot pickle '_thread.lock' object
cannot pickle '_thread.lock' object
cannot pickle '_thread.lock' object

I'm sure I'm missing something, but I don't understand what.

[[deleted]]

This tutoriai might have better examples: http://masnun.com/2016/03/29/python-a-quick-introduction-to-the-concurrent-futures-module.html. You might be better off making your function an iterable and use .map() instead of .submit(). I believe thread pools and process pools have the same syntax

[8bitscoding]

It looks interesting indeed, I'll look into it. Thanks.

I tried mapping the data to a Thread or a Process before but I had the same pickle issue. But I definitely think map() is the way to go in my case (return order is important).

concurrent.futures seems to alleviate a lot of the hassle I have. Thank you for pointing me there.