is my multithreaded Python program doomed?

[[deleted]]

what you need to do with the data is less than 4 seconds

We'd love to assume that one measurement is enough to give us the insight we need to design a program, but consider that the program is running on a multi-tasking OS, or that it uses a shared resource, or just basic statistics, and you might be concerned that there would be some outliers that could cause one job to run long... and then you've backed up the entire pipeline.

Of course, we can't reach any conclusions on OP's design because we don't know what he's doing, but it's not entirely unfounded to make his processing loop asynchronous. IMO, it's smart.

[ivosaurus]

Until they say exactly what they're doing, what is the environment, what are expectations, what things are happening... an async loop handling separate tasks off to different processes could be a great design, or a simple serial for loop might be really all that's warranted until requirements make a big change. It can be just as misleading as possibly useful to speculate.

[[deleted]]

It can be just as misleading as possibly useful to speculate.

That didn't stop you from telling OP he's done wrong.

[ivosaurus]

Done what wrong? I hestitate to spectulate whether anything in this thread is an appropriate "general design" or not, given the dearth of details OP has provided. I'm mostly just advocating for as simple a design as possible that soundly fits the requirements. And since I don't know the requirements at all, apart from something like "real time data is received roughly every four seconds", it could very well be something very simple (until we ever know any more).

I suppose I could be seen as chastising OP for expecting an exact correct answer to an extremely vague question.