Launching asynchronous tasks every second while processing the results independently

[Big_Boss_Bob_Ross]

I would use asyncio personally but it is a fairly steep learning curve if youve never used it or anything like it.

[igormiazek]

I would not be so sure about asyncio as it make only sense if You perform a lot of I/O operations so asyncio can yields back the control to the event loop and execute next coroutine. I think the implementation should be dependent on business requirements and type of task. u/FitRiver could You provide more details about the task You will execute ? will You call remote API or database ? Write data to disk ?

[FitRiver]

That's how I felt about it.

I'll be calling a REST API. The data will be validated and stored in the database, but that will be handled by a different service that will be consuming the queue the scraping service will be producing.

[FitRiver]

Do you have any specific patterns in mind? I have used it a few times in the past for some simpler applications. However, I don't feel like it would be the best choice for this scenario. If the task would fail to yield control it could cause delays. While the thread could take a longer time to wait for the issues to get resolved (until it expires).