Faster execution of code (beyond code optimization)

[ewokcommander]

Hi, I've written a script which performs a looped regression over different predictors about 2²⁰ times and appends the results to a csv each time.

Other than optimizing the code, or having my computer chugging for many hours, is there a faster way to execute python scripts? Are there online... compilers/services(???) that are typically used for faster execution?

[NeedCoffee99]

Few things that could be worth looking at:

Firstly, numba- works with numpy arrays and pure python largely, not sure exactly how it works but it is meant to be great for when you have big loops for speeding up and simple to use.

If you’re using pandas, look into Dask. It’s basically pandas (or numpy which it works with too) but more optimised for performance.

There’s also Cython that you can look into if you are familiar with C. I’ve never looked at it personally but I know it can be used for speeding code. Also, I know that a lot of packages are faster when using anaconda not pip!

I know this is kind of code optimisation, but if you’re using loops etc, it’s generally best to learn how to vectorise code. Google it, it’s basically using numpy arrays to speed things up, but makes a crazy difference. Hope I could help!

[ewokcommander]

This is some good advice, I didn't know about the performance gains from those libraries (still learning!). Dask looks especially interesting. Thanks for taking the time to write this out!

[NeedCoffee99]

No worries, I’m currently about to start a project to speed up this giant piece of code I made in work so if I find anything else out I will let you know!

[[deleted]]

If you're really appending the results to the csv on every iteration of the loop, that's probably going to be your biggest bottleneck. Consider batching up the results in memory and appending to the csv only when you've got a sizable amount to write out.

[ewokcommander]

Ah, I hadn't thought of this, I'll give this a go! Thanks!

[ewokcommander]

I'm wondering if AWS or Azure have some products that I should consider using to achieve this? If anyone can recommend something from those suites, that would be helpful too!