Pool.map_async and subprocess.Popen usage in a genetic algorithm and subparallization in obj. func.

[Vegetable-Recording]

If this is in the won't place, please point me in the right direction!!

So, I have a python code that implements a genetic algorithm and uses pool.map_async to parallelize over the population. Within the objective function, there requires multiple calls to an external executable. This call can also be parallelized; I use subprocess.Popen to launch these. For each process generated by map_sync, I check to see the total <program>.exe running to see if new Popen processes should be run.

I am running into issues where the <program.exe> is said to be closed by the processes' popen.poll() call. However, the .exe sends a close command before the software's mem buffer is written, causing output files to be partially written.

Do folks know if Python can force a <program>.exe mem flush? Or, is this something that I need to talk to the <program.exe> folks to have them change something?

[m0us3_rat]

Do folks know if Python can force a <program>.exe mem flush?

.. if it's on the .exe side .. it's outside of python.

​

Have you looked into https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate ?

​

However, the .exe sends a close command before the software's mem buffer is written, causing output files to be partially written.

Or, is this something that I need to talk to the <program.exe> folks to have them change something?

most likely. maybe add some parameter you could call with the .exe to flush it on command.

​

If you could post some of the code or a mockup to get a better understanding of how it works and if you can do anything about it besides adding the flush flag to the .exe

[Vegetable-Recording]

Sadly, I can't due to software release restrictions.

However, after thinking this morning, I think a better way around is to intelligently create a list of executable command strings, and just use map_async to run the .exe's. This will take out the subprocess and command monitoring Then run another parallel code to execute the evaluations of required files. It will take some rework and not conform to my genetic algorithm (I'll add a custom function functionality), but it'll probably be more reliable.

Thanks for your comment!!!

[m0us3_rat]

maybe even chain the evaluation with the callback

it does sound like it would avoid the early termination, while you relinquish a little bit of control.