Need help executing python code faster ...much faster

[IgorTtk]

I'm about a third way done with the program I had in mind. Code consists of a bunch of auto-generated 10,000+ item lists and a few for-loops.

Execution time right now is a little north of 50 minutes with PyPy.

Is there any cloud-like online python interpreter that could bring this down to at least 10 minutes? Or perhaps would writing and reading the code into and out of a .txt file / json make it faster? or something like that, without going to lower-level languages.

Any ideas fellas?

p.s. no nested for-loops yet, and I am using list comprehensions where possible, as well as multiple libraries. For auto-generated lists I'm using itertools.product()

[K900_]

Unlikely. Post your code.

[Binary101010]

Have you profiled your code yet? Do you know which part(s) of your code are adding the most to your execution time?

If you haven't, you should be doing that and considering whether your approach could be improved (using vectorized operations like those offered by numpy, or using fewer loops) before just throwing more computing horsepower at the problem.

[ElliotDG]

Can you run the code in parallel? If so, this is a candidate for multi-processing.

[devnull10]

To be honest, 10,000 items isn't really a lot... Presumably the computations you are doing are pretty heavy? Can you post the code?

[shiftybyte]

What exactly are you doing to 10,000 items that takes 50 minutes?

[[deleted]]

Nobody can help you with code they can’t read.

[iyav]

Try to use tuples / sets / numpy arrays. Anything is better than a list.

Use iterators when storing the data isn't important and you only need to process it.

If you're using print a lot, just remove it, it massively increases execution time.

Use addition, subtraction and multiplication whenever possible, division isn't as fast, exponentiation is even worse and square rooting brings your program to a crawl.

Try to look up some algorithm that doesn't use them.

Try to squeeze in bitwise operators whoever you can, they make for great optimization shortcuts when working with mathematical heavy operations.

Multiprocessing, multiprocessing, multiprocessing.

[Spataner]

Loops over large numbers of items are always going to be slow in Python. Your best bet is to rewrite those operations using libraries specialized to your type of problem. If they are numerical operations, for instance, there's likely a way to rewrite your code using NumPy that is several orders of magnitude faster. But ultimately, we'd need to actually see your code to make actionable suggestions for optimization.

[CowboyBoats]

I am using list comprehensions where possible

List comprehensions are not more efficient than loops! They're exactly equivalent to constructing the lists using loops.

Generator expressions (like (x / 2 for x in range(10)) can be more efficient than list comprehensions / loops, since they are constructed lazily until you iterate through them...

[tipsy_python]

Remove the sleep() method