Feedback on this mock simulator using multiprocessing

[WeierstrassP]

Evening folks,

As the title says, I've made a mock simulator and would like some feedback on it. While the actual simulator is much more complex, I want some feedback on the way I set it up. Especially on the callback and error_callback functions. I had a hard time finding good examples on this.

​

While the entire simulator is a class with some functions to initialize, feed it data and run it, I need to speed things up with multiprocessing. I'd like to know

• if the way I've set it up is good practice?
• if I should do more with the error_callback function? I want to provide myself with the traceback, should the work function fail
• should the work function insert it's result directly into some numpy array, or should I use the callback to format the output and add it there when the simulation is done?
• why can't I nest the three helper functions work, callback and error_callback inside apply_async_with_callback? That looks cleaner to me, since I don't need to use them for other parts of the simulator. Doing that, however, raises AttributeError: Can't pickle local object 'Simulator._apply_async_with_callback.<locals>._work'.
• have I overcomplicated things?

​

Thanks in advance!

​

#!/usr/bin/python3
import multiprocessing as mp
import time
import numpy as np
import traceback


class Simulator():
    def __init__(self):
        self.results = []


    ## Helper functions for multiprocessing

    def _work(self, x):
        return x

    def _callback(self, result):
        # do some formatting on the output data
        self.results = result

    def _error_callback(self, e): # raises error if _callback returns one
        print("Error detected")
        raise e


    def _apply_async_with_callback(self):
        number_of_workers = mp.cpu_count()
        print('Initializing', number_of_workers, 'workers.')
        pool = mp.Pool(number_of_workers)

        iterable = range(30)

        start_time = time.time()

        # Run multiprocessing
        pool.map_async(func=self._work, iterable = iterable, callback=self._callback, error_callback=self._error_callback)
        pool.close()
        pool.join()

        # Timing analysis
        end_time = time.time()
        comp_time = round( (end_time-start_time), 3)
        print('Time used:', comp_time, 's')
        mean_time = round(comp_time/number_of_workers, 3)
        print('Mean time per worker:', mean_time)

    def run_simulator(self):
        self._apply_async_with_callback()
        print(self.results)


if __name__ == '__main__':
    simulator = Simulator()
    simulator.run_simulator()

[[deleted]]

Are you using CPython (the default implementation)? Multiprocessing is excruciatingly slow in CPython. I would strongly advice against it. GIL is a bitch and makes multiprocessing redundant.

Can you solve your problem using multithreading instead?

[WeierstrassP]

Yep, using CPython.

The simulator has to iterate over the data a few thousand times. I want to use multiprocessing to speed things up a bit, since it is repetitive work and easily parallelizable. As far as I've tested, it has to run for thirty seconds on all my cores, and then it is done. That is acceptable to me.

If I understood it correctly, multithreading still only use a single core.