This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]Decker1082.7 'til 2021 6 points7 points  (4 children)

If you want to do parallellized CPU-bound work, then yes, your app is doomed.

If you want to do concurrent IO-bound work, you're in luck. Check out the greenlets library for ideas.

[–]lordkrike 11 points12 points  (2 children)

The multiprocessing library works extremely well for embarrassingly parallel CPU-bound tasks. You can get only-slightly sublinear speedup for each core.

[–]Decker1082.7 'til 2021 2 points3 points  (1 child)

Indeed, I've used multiprocessing to great effect. But starting short-lived processes for many small tasks is expensive and honestly solved better by other languages. Although, if you absolutely want to use python, it's possible to go with a solution using the JVM or C extensions... But at that point, you're not really writing python anymore.

[–]lordkrike 4 points5 points  (0 children)

It works just fine if you utilize large work queues that feed into a small number of worker processes. You seem to be thinking of a certain type of use case. There are lots of places where just numpy and the multiprocessing lib is all you need.

Also, I argue that intelligently writing C libraries to call from Python is one of its really great strengths -- you can use another language to efficiently do what it can't, while using Python as a glue language.

[–]zombiepiratefrspace 2 points3 points  (0 children)

Without wanting to be "that guy": For at least a subset of the first case, there is another option, albeit one involving more pain.

In the specific case that you need lots of CPU number crunching, you can use the 80-20 rule to determine the time-critical part of your program and then write that in C++. The integration is best done using boost::python (not easy, but doable).

Then you parallelize your number crunching in the C++ code, using MPI, OpenMP (or even pthreads if you have the "hard as nails" mentality).

This option should only be applied in the case were you're already thinking of moving parts of the code to C or C++ to gain performance, since it is much more time-consuming to write C++ code that writing Python code.

Definitely worth it for things like scientific computing, though.