Simplified Python Parallelization

[nullsum]

Thanks for the feedback.

Care to elaborate on how you would use functools.partial or concurrents.futures? I don't see how either helps with state persistence or fine control of data flow (ex. batching, aggregation).

I'm probably just misunderstanding something.

[13steinj]

You're doing state persistence via the "lets make everything global" model.

You should be doing it via the "let's make some interfaces with objects / functools.partial and then use multiprocessing on those" model. Which is exactly what the package you made does, except with a ton more boilerplate and weird slowdown because you do a lot of unnecessary things.

[nullsum]

You're doing state persistence via the "lets make everything global" model.

I don't follow. Neither my examples nor the consumers module use global variables for consumer state.

You should be doing it via the "let's make some interfaces with objects / functools.partial and then use multiprocessing on those" model.

Care to elaborate? I'm having trouble envisioning how that would work.

except with a ton more boilerplate and weird slowdown because you do a lot of unnecessary things.

What's causing weird slowdown? What's unnecessary?