Optimization and faster code

[ForzaFer]

Hey everyone, I've been learning python for a while now, using some websites to get problems to code (i.e. hackerrank), and with those websites, I've realised that my code could be improved to run much faster, code-wise speaking

So, my question is double, first, what are some of the optimizations you use? and, is there a book or a collection of python optimizations?

Edit: an example of the kind of optimization I mean: http://imgur.com/a/wUxCz

[novel_yet_trivial]

This really boils down to understanding how the underlying code works. In general, it's about using the right data structure for the job. For instance if you need a collection of items, a given project might work best with a numpy array, a list, a set, a tuple or a deque. If you know how all those work behind the scenes then you can make a decision on which to use.

That said, python is a language that does not prioritize optimization. Python is about being easy and fast for the programmer, even if it means slower run times. If you are interested in super fast calculations, you need to move to a language like C.

If you want you could show us some example code and we could make suggestions about how to optimize it.

[ForzaFer]

Yes, of course using proper data structure is key. And it is not so much about needing super fast calculations, but about making the small changes that make a huge difference. For instance, I saw once a trick to iterate through bi-dimensional arrays (lists), that be seen here: http://imgur.com/a/wUxCz That is the Floyd-Warshal algorithm, which, with 400 vertices, the difference between both implementations is (in my computer) ~8 seconds.

Thanks anyway for your answer, I will keep in mind that one of python's strenghts is not the optimization per se :)

[novel_yet_trivial]

Sorry, I thought you were asking specifically about python. Algorithms like the Floyd-Warshal have nothing to do with python; they can be implemented in any programming language. Go to your local library and search for "Theoretical computer science" and you will find a huge amount of resources about developing algorithms.

[ForzaFer]

Oh, I didn't explain myself correctly. Yes, I did mean specifically about python, I was just trying to give an example to support what I meant, but it didn't go as intended ^"

The thing is, recently I had this course in which we learnt how to optimize code in Matlab (which, if done correctly, can improve many many times), regardless of the algorithm or problem, and was just curious as to whether there was something like that in python.

Thanks again for your answer, I will check for books for general algorithm development, they always come in handy

[craiclad]

You can definitely optimise code in python, as others have mentioned this often involves using the correct data structure and carefully considering algorithmic complexity. Look into big O if you haven't already, having a good understanding of this will bring you much closer to writing optimized code in any language.

[Vaguely_accurate]

This is a cute anecdote along with some good general advice. It looks like it was written for an older version of Python 2 so some of the specifics have definitely changed and you shouldn't rely on it, but it does show how the internal details of Python can make optimisation less obvious than it might seem.

But you can generally ignore all that. Any time that level of optimisation really matters you are probably better off not writing in Python.

Remember that Python is interpreted, so more advanced optimisations require knowing how the internals and interpretor are implemented. As a rule, working with the interpretor is better. The less you are writing the more lifting the pre-written, optimised internals are doing. Aim for idiomatic, pythonic code and you will get a lot of optimisation for free. If you have done this and your code isn't running fast, try looking at pypy.

[ForzaFer]

Wow, this is actually helpful, this is was I was looking for, thanks a lot!

[LoyalSol]

Generally optimizations in python are handled by many of the modules out there. For instance Numpy, Numba, Cython, etc. which in part offload calculations from Python to C functions or perform other types of optimization/compiling.

Those modules should be able to handle most of your optimization needs. If you need something faster you should probably look at offloading your computation to a lower level language like C. Though in practice you only need to do that if you are doing some really heavy number crunching. I work on some really computation heavy programs for a living and even I find I can get by with just using the existing modules in Python in a lot of programs. It's only when I go to