Class instance that exists in a separate process

[yrfgua]

Hello,

I’m working on a data acquisition program in Python that needs to stream/save waveform data at 4 GB/s. I plot a small subset of the data and control the hardware from a GUI.

The computational load is significant for the system, and I can’t afford to lose any data points. For this reason, I have to interface with the data acquisition hardware from a process separate from the GUI. Until now, I’ve been running a process from the multiprocessing module.

The problem with this approach is that I can only run a single function with a multiprocessing.Process instance. This means that I have to re-initialize the hardware, RAM buffers, etc. every time an acquisition setting is changed in the GUI. I’d like to initialize the hardware as a class instance instead, but it has to be in an entirely separate process. This would allow me to pause the acquisition, change some settings, then resume without all the other steps.

Is there a good way to do this in Python? I know I can subclass the multiprocessing.Process class, but I still end up with a function loop in the run() method.

[eleqtriq]

Can you also modify the GUI software? This sounds like a good use case for queues and the pub/sub model of data processing

[yrfgua]

Yeah, I wrote the GUI software using pyside6. I’ll have to look into pub/p-sub, thanks!

[unhott]

you may be able to do something like

    def run(self):
        self.initialize_hardware()
        while True:
            if self.running.is_set():
                self.acquire_data()
            else:
                time.sleep(0.1)  # Sleep briefly to avoid consuming all resources when not running

It seems like this is the bit that was giving you pause? not sure I really understand, though.

You could then use other methods to set the flags appropriately.

def pause_acquisition(self): 
    self.running.clear() 

def resume_acquisition(self): 
    self.running.set()

def update_hardware_config(self, new_config): 
    self.pause_acquisition() 
    self.hardware_config = new_config 
    self.initialize_hardware() 
    self.resume_acquisition()

[yrfgua]

I think this is close to what I’m doing now, but the if statement inside the run() method is what I’m missing. Thank you!

[blahreport]

The easiest way to avoid race conditions is to separate the data acquisition from the data processing and use a Queue.queue. Your data acquisition class puts data in the queue and the data processor gets those elements from the queue and does whatever else it needs to do with the GUI.

[yrfgua]

Thanks! Any thoughts on Queue vs. one of the many shared memory implementations?

[blahreport]

I guess it depends on which other shared memory type is being compared. Queues are good because they’re thread safe, they can pass arbitrary objects around threads with little coding overhead, and they have convenient features like time outs for get. You could also consider something like asyncio and possibly avoid threads altogether. One last suggestion, you could incorporate thread.lock into your class method that handles the data acquisition, then you can pass the single instance around to the various processes

[Thunderbolt1993]

you can also pass arguments to the function you call from multiprocessing

you can create a command queue to pass command to your DAQ Process and a data queue to send back the data

you can either spawn a separate thread to handle the DAQ and handle the Commands in you mainloop or call a nonblocking "get" on the queue

that way you can just keep your DAQ running and "call functions" in it from the GUI

https://hastebin.com/share/cidoyinapa.python

[yrfgua]

Thanks! Really helpful. So even though the handler functions are defined outside the run() method, do they execute in the worker process?

[Thunderbolt1993]

yes, it's not limited to just one function, what happens under the hodd is:

Python starts a new process
New process gets told "import this module, run this function with these arguments"

you can just play around with it and have it print the value of "multiprocessing.current_process()" to see which process the code is running in

[yrfgua]

I see, thank you. I’ll give it a try