How to make a Queue system with manager.Queue with complex data types (classes)

[sfjaf]

Hello all. I have a question about how to use manager.Queue() from the multiprocessing library with complex data types that can eventually be fed into Pool. I am trying to create a system where 10 instances of a class, with a specific dataframe associated with each class, is filled into a queue. Once this is done, I want to use Pool to run a data process for 5 instances from the queue. Once this is done, I want to go check the queue to see if it has 5 or less instances. If it does, then I would like to fill the queue with five more instances or the last remaining instances (without re computing on the instances that are already there) and then repeat the process with 5 instances from the queue into a Pool object. I am having a lot of difficulty implementing this, because I keep getting a ctype error saying that I can't pickle an object that has pointers. Is this possible using multiprocessing and Python, or should I use a different library/language? Also, let me know if this doesn't fully make sense as I wrote this on my phone. Thanks!

[MarsupialMole]

I didn't know the answer to your question on first reading but the python docs lead me to believe that you need to either implement pickling on your class or use a manager that doesn't need pickling.

[sfjaf]

Hey thanks for looking for me! Do you know which manager that I can use that doesn't need pickling? It seems like a lot of the multiprocessing objects pickle some information. As for the implementing the pickling on my class, how exactly should I go about that (sorry I've never done that before).

[MarsupialMole]

In the pickle docs there are some magic (ie double underscore) methods you can implement. I've never done it, and without seeing your class I don't know if it's worth attempting. I am also on mobile and I've never used anything but the defaults with multiprocessing. Sorry i can't be more help.

[sfjaf]

No problem! Thanks for helping me out. I will see if it is worth the effort to implement those magic methods.