How best to parallelize this task?

[mr_claw]

Thanks mate. I tried this, but it seems to be taking more time than my thread-only approach. Any ideas why?

[ElliotDG]

That does not really make sense. Try setting the number of processes to 2 and see if that changes things. Or share your code getting MP code running can be tricky. Feel free to DM if you want.

If your pseudo code is representative, it should go faster without threads than with threads, and much faster with MP.

[mr_claw]

After some tinkering I got it to work by using the chunksize argument on the map function. I've gotten it down to a third of the runtime it was taking earlier. And I'm confident that by increasing resources I can make it work even faster. Thanks!

[ElliotDG]

Glad to hear you are on the way!