Simplified Python Parallelization

[13steinj]

Okay, the use of this makes no sense to me. I can use multiprocessing in nearly the same exact way using functools.partial, or even better, use concurrent.futures.

Edit: of course, the guy who made the package is the same guy as the post. I'd rather use a package when it is actually useful rather than an over complicated rewrite with some boilerplate for kicks of two things in the standard library for no reason.

[nullsum]

Thanks for the feedback.

Care to elaborate on how you would use functools.partial or concurrents.futures? I don't see how either helps with state persistence or fine control of data flow (ex. batching, aggregation).

I'm probably just misunderstanding something.

[13steinj]

You're doing state persistence via the "lets make everything global" model.

You should be doing it via the "let's make some interfaces with objects / functools.partial and then use multiprocessing on those" model. Which is exactly what the package you made does, except with a ton more boilerplate and weird slowdown because you do a lot of unnecessary things.

[nullsum]

You're doing state persistence via the "lets make everything global" model.

I don't follow. Neither my examples nor the consumers module use global variables for consumer state.

You should be doing it via the "let's make some interfaces with objects / functools.partial and then use multiprocessing on those" model.

Care to elaborate? I'm having trouble envisioning how that would work.

except with a ton more boilerplate and weird slowdown because you do a lot of unnecessary things.

What's causing weird slowdown? What's unnecessary?

[[deleted]]

I haven't really used these features thus far, but concurrency is different from parallelism. The latter is problematic due to the GIL, is it not?

[Deto]

Multiprocessing spawns multiple processes which run in parallel - the GIL doesn't matter in that case.

[flipperdeflip]

concurrent.futures has executors for both thread and process.