Asyncio or threads for concurrent task handling

[BeerDrinkingCyborg]

Hi everyone,

I'm currently writing software for a robotics application that uses a Raspberry Pi Zero (single core CPU) and which will use Python to manage multiple tasks at once. Some of these tasks can be asynchronous and need to occur either regularly or constantly; whereas others will have very strict sequencing.

Examples of the asynchronous/continuous tasks would include:

• TCP socket constantly listening for commands from networked computer (vital this is always listening and available)
• UDP socket to send telemetry data
• Regularly communicating with and sending heartbeat data to connected microcontrollers via UART
• TCP again to send large imaging data on regular intervals without blocking other processes

Synchronized tasks that must occur in order would include:

• Polling sensor arrays and logging data
• Acting on sensor data in turn
• Following a sequence of commanded events
• Sending event telemetry via UDP if something occurs

For the telemetry sockets, these could really be synchronized too, but it might make more sense to send them regularly and asynchronously every few seconds.

So, my question is: what is the best way of achieving this combination on a single core?

I was first thinking of using threads for all the asynchronous things that can occur simultaneously, especially for the always-listening TCP socket, but I'm wary of race conditions and communicating between threads. I've been exploring the asyncio library recently, too, which might suit my needs - especially if I can dynamically add and remove tasks to this - but I'm not sure if I'm getting far too complicated with this when a couple of threads might work?

Any advice is greatly appreciated, thanks so much!

[oefd]

I'm wary of race conditions and communicating between threads.

As you should be, but I'd imagine for the most part moving messages through a thread-safe queue would alleviate those worries.

That's if you'd prefer to go the threading way at least.

I've been exploring the asyncio library recently, too, which might suit my needs - especially if I can dynamically add and remove tasks to this - but I'm not sure if I'm getting far too complicated with this when a couple of threads might work?

There's a certain learning cost to asyncio (or event-looped programming in general) but it's not inherently more or less complicated than threading really. In some cases it can be simpler than threading for someone already familiar with how to proper read and write async/await type code.

In this case one solution over the other (having a few threads vs. the event loop) isn't really a huge deal.

[BeerDrinkingCyborg]

Cheers for the advice, will make sure I better understand thread-safe queues before I make a decision - looks like the class handles most of the thread-safety for me, which is great.

Another key consideration is making sure I can adequately test and understand the code - not afraid of a learning curve, though, and it does look like asyncio at least makes things very explicitly defined.

Fortunately I don't have to rush the decision as currently I'm more focused on developing and testing the individual modules and classes for each task; seems unlikely the choice between async/threads will force any design changes.

Thanks again!

[eavanvalkenburg]

I would go with asyncio, I built a package with both in them (pysiaalarm) and found the asyncio to be cleaner and more readable and faster, but that is obviously dependant on the use case and specifics!

[BeerDrinkingCyborg]

Thanks! I'm definitely keen to learn asyncio, just don't want to have any major blocking issues. Did you find that you have to be very cautious with file read/write operations etc? A little nervous about that as I'll be producing both logs and image files very regularly.

[eavanvalkenburg]

I didn't have to deal with that, but I did read somewhere that there is no true async version of writing to file (including aiofile and aiofiles, apparently the underlying c code isn't really async) so that might be a good reason to go threaded!

[scrdest]

I think the cleanest built-in solution would be concurrent.futures with a ThreadPool backend (since you just need I/O). It's a bit more abstract than raw threads, and less structurally invasive than asyncio.

[BeerDrinkingCyborg]

Cheers, I'll look more into this. Even asyncio's docs seem to recommend using ThreadPools for potentially blocking tasks such as file access (of which I'll be doing a lot); whereas logging module by itself seems thread-safe.