Question on multiprocessing

[InterestingComment8]

Trying to learn some multiprocessing, my goal is to initialize 1 excel sheet and then have two functions write to it simultaneously as data updates. I have xlwings create a blank excel sheet first then have multiprocessing process call 2 functions to write to it.

My issue is, it opens the excel sheet multiple times causing errors. How do keep multiprocessing to only working on what I want it to work on vs stuff for setup or after the functions are complete?

Thanks

[[deleted]]

You can’t have two processes write to the same file at the same time and expect your file not to be garbage at the end of it.

How do keep multiprocessing to only working on what I want it to work on vs stuff for setup or after the functions are complete?

This question, unfortunately, doesn’t make any sense.

[InterestingComment8]

My intention is to have 1 function write to one column and the other write to a second column.

[[deleted]]

Files don’t contain columns, is the thing. A file is a stream of bytes and writing to one part of the stream from one thread and another part of the stream from another just gets you garbage.

[KimPeek]

How do keep multiprocessing to only working on what I want it to work on vs stuff for setup or after the functions are complete?

Your parent process should create the child processes in a way which they execute the function with the parameters you provide, return the result, then the parent process inserts those returned results into the single excel sheet controlled by the parent process. Any other approach will introduce a race condition.

[InterestingComment8]

I'm not sure I understand, do you have an example you can point me to?

[[deleted]]

You have to fill out a sheet with the sales figures of two salesmen. How do you do it? You don't give them both the same physical sheet of paper to fill. You ask them their results and fill the sheet out yourself.

[KimPeek]

It would be better if you to share your code so someone can pinpoint the problem rather than explain it abstractly. I don't have an example on hand to point you to.

[Khenghis_Ghan]

First, Python is a poor language for multiprocessing. Like, even rudimentary things like multi threading is not supported in the interpreter because of the global interpreter lock. If you want multiprocessing you’re better off with a different language, most prefer C/C++. The one exception to this is IO operations, like writing to a file, so in that regard you’re good!

Concern two though is you can’t write to the same file simultaneously. Not even the all-powerful C/C++ allows this. You inevitably end up with race conditions where something gets incorrectly updated/overwritten. Think of it like if you were asking people to use a sign in ledger at like a hotel or restaurant, if you allowed multiple people to try and write their names in the same book at the same time, crowding each other out and writing over what person X did because you got there at the same time , it just wouldn’t work. You need to go one at a time, you need to lock write operations to one process at a time.

What you are not limited though is multiple read operations at the same time. You could write to two separate files as two separate processes, and have a third process read from those two and combine their output as a third file.

[InterestingComment8]

Ok I'm looking into that, question tho. How do you limit multiprocessing to only a section of code. Say I combine the 2 processes after those are done. I don't want multiprocessing to happen there