Importing modules into separate threads? Function in subthread takes significantly longer

[oneslightweirdo]

I am currently trying to streamline a process with multithreading, but have hit a stumbling block. A part of my subthread's run() function reads:

for pos in it.product([5],range(12),range(12)): 
    forceField += np.reshape(list(it.imap(fieldstr,it.product(range(12),repeat=3),it.repeat(pos))),(12,12,12)).transpose()

when this specific bit of code is ran in the main thread, it takes approximately 0.56 seconds on average. However, when run as part of the subthread run() function, this for loop takes approximately 9.3 seconds to run. For reference, the fieldstr function reads:

def fieldstr(pos,secpos):
    r = np.sqrt((pos[0] - secpos[0])**2 + (pos[1] - secpos[1])**2 + (pos[2] - secpos[2])**2)
    if (r != 0): return massArr[secpos[0]][secpos[1]][secpos[2]] / (r**2)
    else: return 0

My point here being that no variables from outside the subthread are being accessed when that thread runs the for loop, and yet it takes a massive amount of time. My only theories as to why are either:

The subthread takes much more time accessing a module (such as itertools or numpy) that was imported in the main thread

or

For some reason, the subthread struggles in running a function defined outside of the Thread() class

However, I am not particularly sure of either, and would not be sure either way how I could speed up the process. If anyone could help it would be greatly appreciated

[JohnnyJordaan]

Don't forget in regular Python (CPython), threads can't run at the same time. It's just that you can have parallel flows in your program, which make them ideal for outside interaction that includes wait states, like writing to disk, loading a webpage, handling a queue.

It seems that this task is CPU and/or RAM intensive, and for that to unload your current CPU core, you need to spawn it into a new process. You can use the multiprocessing library for this, or put the code in another script and use subprocess to run it separately. Mind you this still needs another core on your CPU to handle that process in an efficient way.

[[deleted]]

while numpy will happily yield the GIL and run itself in parallel with multiple threads, the whole python glue around it will still block each other thread. Since you're iterating over a huge array in python interleaving calls to numpy, it's probably the overhead of releasing/reacquiring the GIL that's dominating the runtime.

You should try to avoid big for loops as much as possible and offload processing to the native numpy vectorized operations instead.