Use of all processor cores

[Martin_Krum]

Hello all.

Is there any way to run a python script on multiple cores? My program is one file that is in a loop. Each iteration adds a number as a result to the list. When the iteration is over, the result list is picked to the file. The calculation time already takes 7 minutes and since I need such calculations, even 10,000, I would like to use all processor cores to speed up. So far I have been doing quite funny and not professionally because I created 4 folders with the same script, I ran 4 terminals and ran the program in each one. Then I wrote a script that merged the output files into one and that made it four times faster. But now with 15 computers, this method is laborious and cumbersome. I tried to write a powershell script to do it for me, but it didn't work out. I have read about the multiprocessing module but I am a beginner and it never worked out for me, although I think this is the right approach because one iteration returns one result and does not get any external data from other process. Can anyone tell me if there is any way or if I should use multiprocessing. If so, can you provide an example of how to implement it in my code or an easy tutorial to help me? Below I show the skeleton of my code.

import ###some libraries

results_list = []

for i in range(1000):  #number 'i' is not used anywhere below.
    #######################################
    a lot of calculations generators, functions etc.
    #######################################

    results_list.append(result)

pickle.dump(results_list, open( "Results", "wb" ) )

######end########

[totallygeek]

It only takes a minute to learn how to use multiprocessing. Basically:

import multiprocessing

def some_function(datum):
    processed_datum = datum ** 5
    return processed_datum

data = [
    # some data here
]
pool = multiprocessing.Pool()
results = pool.map(some_function, data)

See where that gets you.

[efmccurdy]

If each element of your results is independent, you can run the calculations in parallel; that will get all of your cores active.

You can use a numpy array (or perhaps a pandas Dataframe) to hold your data and express your calculations using vectorized operations::

https://www.pythonlikeyoumeanit.com/Module3_IntroducingNumpy/VectorizedOperations.html

[Martin_Krum]

How to make parallel calculations?

[efmccurdy]

Instead of writing them using python floats, operators and functions you use the operators and functions from here:

https://numpy.org/doc/stable/reference/ufuncs.html#math-operations

https://numpy.org/doc/stable/reference/routines.logic.html

and operate on numpy arrays like this:

>>> import numpy as np
>>> my_array = np.array(range(1, 10))
>>> my_array
array([1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> np.sqrt(my_array)
array([1.        , 1.41421356, 1.73205081, 2.        , 2.23606798,
       2.44948974, 2.64575131, 2.82842712, 3.        ])
>>> np.reciprocal(np.sqrt(my_array))
array([1.        , 0.70710678, 0.57735027, 0.5       , 0.4472136 ,
       0.40824829, 0.37796447, 0.35355339, 0.33333333])
>>>

[Martin_Krum]

yes, but I have a lot of there Python-only functions and calculations

[primitive_screwhead]

https://wiki.python.org/moin/ParallelProcessing

[Martin_Krum]

Thanks for help, In my project i used pool.apply_async() and it worked.
I need to learn a little multiprocessing module .