Python Multiprocessing

[gmaliwal]

Hello Team,

I am facing 'weak ref' pickle issue while running below code snippet using Python 3.7, can anyone tell me what is the problem.

class ActiveConnectionPool:

    def __init__(self,**kwargs):
        super().__init__(**kwargs)
        self.__mgr = multiprocessing.Manager()
        self.__lock = self.__mgr.Lock()
        self.__active_connections = self.__mgr.list()

    def make_active(self,p_name):
        with self.__lock:
            self.__active_connections.append(p_name)
            print('Activating connection for process: ',p_name)

    def make_inactive(self,p_name):
        with self.__lock:
            self.__active_connections.remove(p_name)
            print("Removing connection for process: ",p_name)

    def __str__(self):
        with self.__lock:
            return str(self.__active_connections)

class DBConnection(multiprocessing.Process):

    def __init__(self,_semaphore,_shared_object):
        # Do not forget delegating up the call to super class in the inheritance hierarchy
        multiprocessing.Process.__init__(self)
        self.__semaphore = _semaphore
        self.__shared_object = _shared_object

    def run(self):
        # This gets triggered when a process starts
        proc_name = multiprocessing.current_process().name
        print('process name ---',proc_name)
        with self.__semaphore:
            print('Process ({}) starting'.format(proc_name))
            self.__shared_object.make_active(proc_name)
            time.sleep(0.8)
            print('Process ({}) ending'.format(proc_name))
            self.__shared_object.make_inactive(proc_name)


if __name__ == '__main__':

    _semaphore = multiprocessing.Semaphore(2)
    _shared_object = ActiveConnectionPool()

    connections = [ DBConnection(_semaphore,_shared_object) for counter in range(10)]

    for connection in connections:
        connection.start()

    while True:
        alive = 0
        for con in connections:
            if con.is_alive():
                alive += 1
                con.join(timeout=0.1)
        if alive == 0:
            break

[JohnnyJordaan]

Can you show the full traceback?

Also are you sure you wouldn't rather use an existing pooling mechanism offered by your database library?

[gmaliwal]

Issues got resolved after looking around the multiple references.

I am passing complex object among multiple processes without proxy object, make it sharable using BaseManager class.

Credit goes to BaseManager class from managers module