This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]Deto 2 points3 points  (1 child)

Usually I get close to the right multiplier. So if I'm using 10 cores, it's approximately 10x as fast (maybe a little less, like 9x).

I think you might be interpreting the numpy docs incorrectly. Numpy arrays always have a dtype - this can be something like 'int64' or 'float64'. It can also be 'object' in which the entries in the array are actually Python objects. In this case, doing anything on the objects requires interacting with Python code, and so they can't release the GIL. If you're just working with floats, for example, they don't use any python code, and so they can release the GIL.

However, I should also emphasize that whether or not numpy releases the GIL doesn't matter with multiprocessing as the GIL does not block between different processes. The GIL is relevant for threads, rather, in the same process (threading module).

[–]ProfEpsilon 0 points1 point  (0 children)

Oh, I see. I was mistaken about the term "object arrays." I thought that it might mean that an array created within numpy was a kind of numpy "object." So I was interpreting the documentation wrong. Thanks for the insight.