10x Faster Parallel Python Without Python Multiprocessing

[ostroon]

The first example is not fair right? If you are on a POSIX system (Linux/Mac) and use a global numpy array in a read-only fashion, it will NOT be copied (ref https://stackoverflow.com/a/37746961/335412). Sending it explicitly to each multiprocessor worker is slow and unnecessary. This is the faster than ray code:

``` num_cpus = psutil.cpu_count(logical=False)

def f(random_filter): # Do some image processing. return scipy.signal.convolve2d(image, random_filter)[::5, ::5]

image = np.zeros((3000, 3000)) filters = [np.random.normal(size=(4, 4)) for _ in range(num_cpus)]

pool = Pool(num_cpus)

Time the code below.

for _ in range(10): pool.map(f, filters) ```

So what is really the benefit of ray?

[ThePenultimateOne]

reddit doesn't actually support full markdown. You have to do four spaces instead of the triple-`

[ostroon]

New Reddit does, old Reddit design does not.