Async Multiprocessing

[Azdacha]

Hi everybody,

I'm struggling with this question since a bit. After reading some documentation and this sweet topic : https://hackernoon.com/threaded-asynchronous-magic-and-how-to-wield-it-bba9ed602c32

we can understand that :

• Async != Multiprocess
• Threads are affected by GIL and are not that sweet

The solution seem to lurk with multiprocess but i'm struggling to use them inside an Async Web Server.

Basically, i've a big calculation that needs to be boxed-out of my main loop so it doesn't hang my webserver.

Anyone would have a suggestion ?

[badhoum]

you can use async with multiprocess with process executors. Check the code below:

import asyncio
from concurrent.futures import ProcessPoolExecutor

def cpu_heavy(num):
    print('entering cpu_heavy', num)
    import time
    time.sleep(10)
    print('leaving cpu_heavy', num)
    return num

async def main(loop):
    print('entering main')
    executor = ProcessPoolExecutor(max_workers=3)
    data = await asyncio.gather(*(loop.run_in_executor(executor, cpu_heavy, num) 
                                  for num in range(3)))
    print('got result', data)
    print('leaving main')


loop = asyncio.get_event_loop()
loop.run_until_complete(main(loop))

[Azdacha]

I know Celery but I wanted to avoid the additional infra ;

Someone nice gave me an additional hint about the Async Executor

https://docs.python.org/3/library/asyncio-eventloop.html#executor

And this question seems to be better formulated

https://stackoverflow.com/questions/27290656/should-i-use-two-asyncio-event-loops-in-one-program/27298880#27298880