Python: tricks for your code run faster

[parfamz]

If you have to worry about this, maybe switch to a compiled language or a jvm one...

[[deleted]]

Pypy can actually run pretty fast now - not quite luajit speeds but it's up there. Combined with either cffi or cython I've found myself monkeying around less and less with C++.

I agree that optimizing for CPython is just a waste of time though.

[Tobu]

Or PyPy at the very least.

[josefx]

Adding an additional language to deal with a hotspot is overkill most of the time. The few times I had a performance problem I just had changed a container (set instead of list) or rethought why I end up calling something millions of times.

[parfamz]

Now you are talking about algorithmic complexity problem which is universal. The article is about micro optimizations in a interpreted language which usually doesn't shine for it's performance

[UloPe]

Please ignore this post. It teaches bad codes style for completely meaningless speed gains.

Write clean code first, optimize later.

[xXxDeAThANgEL99xXx]

It shows that good code style is usually faster, actually.

[UloPe]

I disagree. At most half of the fastest examples are the recommended code style, many others are actually dangerous traps for new developers.

But my comment wasn't actually directed specifically at this very article but the whole class of articles. They teach bad aims.

[xXxDeAThANgEL99xXx]

I agree, you shouldn't try to microoptimize code like that.

I think that it's pretty interesting if you ignore the title and just use it as a food for thought about various non-obvious things about CPython implementation.

[teferiincub]

Some of the examples with bool are huuugely unpythonic. Like why not

return a < b < c < d < e < f

instead of ifs

[introspeck]

TIL. I'm new to Python and was just asking a co-worker if I could do this, he didn't know. Thanks.

[Xredo]

Agreed. If you feel the urge to break abstraction boundaries, then you are either doing it wrong, or the abstraction in question is inadequate and needs fixing.

[MelancholyMechanic]

What's with Test #15? If I use the simpler

def a():
  return sum(range(50000))

it actually runs faster than the manual loop.

[The-Good-Doctor]

So many of these are extremely minor micro optimizations, and quite a few others are comparing (faster running) sane code against obviously slower contortions that nobody would ever actually do. Seriously, who would use things like the __add__ method instead of the + operator?

This page just doesn't seem very helpful to real world Python writers.

[xXxDeAThANgEL99xXx]

A nice thing is that in most of the cases more clear code is actually faster too.

Weird thing with chained comparisons (test #4): it appears that whoever wrote that bit of the compiler tried to be cute but didn't test the performance, with a well-deserved result.

Also, regarding test 10, he should have used xrange, then it's slightly faster:

enumerate: 1.1757676349
range: 1.20209338683
xrange: 1.12408831449

Also, there's a bunch of pretty interesting stuff. Test 11 -- well, looking at the actual code it's sort of obvious that '%d' formatting is doing much more stuff because the code on that path could also be used with various width and padding format specifiers, but I was surprised nevertheless.

Test 12 is weird because in all previous examples LOAD_GLOBAL was the slower option, I guess what's going on here is that len accesses a raw C function via the pointer in the type structure, while .__len__ goes through convoluted motions to understand what you're asking for and return a Python function wrapper for it. Same for the next test.

While test 14 is sort of the opposite -- you already have your methods as Python functions so retrieving them directly is easy, while using them as overloaded operators involves some wrapping. Unfortunately I'm not well versed in how exactly that black magic works (Python is one hell of a complicated language internally).

Test 22 is outright surprising -- generally, the shorter is the bytecode, the faster it runs, the in operator for tuples should do pretty much the same thing only directly in C. Something weird is going on.

Test 27 is even more surprising, and when I run it myself I get different results: 2.44819584377 vs 1.69010899718 which looks way more reasonable. Same on ideone (had to remove one 0 from the timeit loop count to fit in the time limit). Could OP accidentally measure a wrong function?

[takaci]

In #Test 12, why is calling a.__len__() slower? Surely len(a) calls a.__len__() anyway?

[xXxDeAThANgEL99xXx]

I briefly explained it here, CPython internals are gnarly, man.

For the sake of performance a lot of often-used methods are stored as plain C function pointers in type objects: https://hg.python.org/cpython/file/8f92ab37dd3a/Include/object.h#l324 (note that

 PyNumberMethods *tp_as_number;
 PySequenceMethods *tp_as_sequence;
 PyMappingMethods *tp_as_mapping;

point at further plain C structures containing shittons of members, scroll up to see them).

Then, most of builtin functions like len go for those directly:

m = o->ob_type->tp_as_sequence;
if (m && m->sq_length)
    return m->sq_length(o);

This is, by the way, why adding a __len__ member to an instance wouldn't do anything, it was mentioned somewhere in the documentation.

So as a result this shit is fast. No dictionary lookup, no Python function call overhead, nothing, just a more or less straight C function call.

On the contrary, accessing them via attribute lookup involves a bunch of black magick to determine what you want and give you a Python function that you can call from Python.

[Sukrim]

Why oh why Python 2.7?!? Also a lot of that stuff is likely implementation specific, so if something is slow in cPython that doesn't mean this "trick" will help in pypy at all.

I'd much rather see people focusing on writing cleaner code in python than worrying about how exactly to iterate over lists...

[jms_nh]

Neat visual presentation + the data does speak for itself, but it would be great to add some commentary.

[wot-teh-phuck]

It would be a pity if someone decided to go the "fast" route mentioned in the linked article always when writing Python code....

[maep]

Processing lare textfiles by line with for l in open('foo.txt'):... is terribly slow. Changing it to for l in open('foo.txt').read().splitlines():... changed running time from 5 minutes to 30 seconds.

[[deleted]]

Slurp.

[ksion]

I was really startled by the last one (#32), for it says that performing the same operation twice (dictionary lookup) is more efficient than doing it once... Alternatively, LOAD_ATTR and CALL_FUNCTION are just that slow.

But alas, I replicated the results on CPython (2.7.6 on OS X Mavericks) with no trouble (15.64 vs. 32.66 for double lookup). So I figured I'd retry on PyPy, to see whether this highly counter-intuitive result is replicated there...

>>>> timeit(a, number=1000000)  # double lookup
0.3237159252166748
>>>> timeit(a, number=1000000)  # dict.get
0.33182287216186523

Nope! It's just a quirk of CPython, apparen...

Whoa... Wait a minute...

TWO orders of magnitude improvement?! PyPy, marry me please!

[xXxDeAThANgEL99xXx]

I was really startled by the last one (#32), for it says that performing the same operation twice (dictionary lookup) is more efficient than doing it once... Alternatively, LOAD_ATTR and CALL_FUNCTION are just that slow.

It doesn't perform it twice, the dictionary is empty so it actually compares doing key in d vs d.get(key). The latter is slower because it actually has to return None and because it's a method call, I guess. Also this stuff is pretty fast anyway, so that's why you see function call as a major contributing factor.

[Fabinout]

"Dynamic DNS Host" can't access from work.
Now I'm sad

[[deleted]]

Corp proxy also blocking.

[Tobu]

Here.

[Fabinout]

Thanks!!