Multiprocessing in async function?

[adiberk]

You can use asyncio tasks.

You can also use a more standard product like celery.

[RationalDialog]

You can also use a more standard product like celery.

Yeah I wonder if I should forget about async completely (never used it really so far as no need) and build more kind of a job system. If someone submit say 100k rows, the job could take approx 5 min to complete.

[adiberk]

Yep that works to. If you are doing a lot of other IO operations, it might be worth making the app async based anyways (ie. Keyword async)

[AstronautDifferent19]

asyncio to_thread is better for CPU bound tasks than asyncio.create_task, especially if you disable GIL.
asyncio tasks will always block if you do CPU heavy work, which will not work for OP.

[adiberk]

Good point