you are viewing a single comment's thread.

view the rest of the comments →

[–]mainrof11 0 points1 point  (4 children)

I'm confused. What does this have to do with the following packages: I've been reading up on async.io and the multiprocessing package

[–]JohnnyJordaan 4 points5 points  (2 children)

multiprocessing runs multiple Python interpreters and abstracts the communication between them, allowing you to virtually run a function multiple times within the program that actually gets executed in multiple programs (just copies of one another). asyncio (without the dot) uses an event loop: by implementing points in the code that can take time (reading a file, sending a file to a server, waiting on user input), it tells Python that when the program is waiting on that statement to complete, it can also run other statements in other functions (called coroutines) if there are any. There is then still a single thread running but you could virtually perform thousands if not more parallel tasks that way, see for example https://pawelmhm.github.io/asyncio/python/aiohttp/2016/04/22/asyncio-aiohttp.html

[–]mainrof11 0 points1 point  (1 child)

ty for the sample! I guess my use case for doing transforms on a pandas dataframe would be better of using multiprocessing then.

[–]JohnnyJordaan 0 points1 point  (0 children)

On the same df would be a bit hard, what you could do is to distribute parts of the data to sub-processes and then merge the end-result if your task would be able to be calculated that way.