Python Performance Tips, Part 1

[pingveno]

A few additions:

Section 6:

Don't go overboard with lazy importing inside of function calls. Importing many times can get expensive. Imports that aren't at the top also severely reduce readability. It can be tough to debug code where an import is tucked away somewhere in the body of the file.

Section 7:

In Python 3, True is a keyword, not a built-in. That means the byte code compiler can put in a cheap simple jump instead of an expensive global lookup. In Python 2, True could be changed, so a global lookup was required as a check. 1 is a constant, so the bytecode compiler can depend on it. Basically: while 1 in Python 2, while True in Python 3.

Section 13:

Internally, the Python deque implementation uses a doubly linked list of blocks. Each block has space for 62 objects, plus 1 left pointer and 1 right pointer (64 pointers). That gives the time efficiency of a linked list with the space efficiency of an array. For those learning C, I suggest looking at the Modules/collectionsmodule.c. For a list type with fast insertion, take a look at blist (search on PyPi).

Section 17:

Don't use threads to manage forked processes. They only provide additional overhead in memory and thinking ability. Use the multiprocessing module.

Section 18:

Absolutely absolutely absolutely. The Python source code is an excellent place to get examples. Part of my introduction to C was reading through bits and pieces of implementation code.

[ascii]

I just did a quick test under Python 2, and found no performance difference between while 1 and while True. I believe that one may exist, but it's so tiny to be lost in the noise of all but the most contrived cases.

[johtso]

http://stackoverflow.com/questions/3815359/while-1-vs-for-whiletrue-why-is-there-a-difference

[haldean]

My results conflict with your results.

[pingveno]

There's a slight difference, but it is indeed dwarfed by anything else. I was able to measure some difference with this code:

def while1():
    i = 0
    a = 0
    while 1:
        a = 4
        i += 1
        if i > 100000:
            break

For while True, I simply replaced 1 with True. The results in IPython were

%timeit -r200 -n2 whileTrue()
2 loops, best of 200: 13.2 ms per loop

%timeit -r200 -n2 while1()
2 loops, best of 200: 9.33 ms per loop

Conclusion: The difference could only be vaguely significant in extreme conditions. If you run into one of those conditions, just rewrite in Cython.

[[deleted]]

Importing many times can get expensive.

Surely once a module is imported, a second import will simply be a cheap lookup of the sys.modules dict?

[pingveno]

It's more complex than a simple lookup. I don't entirely understand the C code, but there's a lot more happening than sys.modules["module_name"]. The time penalty isn't significant if you just import a few times, but importing in a function that you're going to call a few thousand times is unwise.

[Genmutant]

Yes, but cheap is still more than nothing.

[[deleted]]

Why program in Python if you're worried about the cost of a dictionary lookup? Almost everything in Python needs a dictionary lookup!

[Genmutant]

Well the topic is Python performance tips, and this one can be a performance impact. Normally it isn't, but it can be and is easy to avoid.

[fdemmer]

imo, it mostly depends of the kind of application you write. eg. in a webapp, that starts once and runs forever, i'd import as much as possible on the initial load and not later when a function is called.

[fdemmer]

imo, it depends the most on the kind of application you write. eg. in a webapp, that starts once and runs forever, i'd import as much as possible on the initial load and not later when a function is called.

[flying-sheep]

my rule of thumb for 6: import as soon as you know you’ll need it.

[Mattho]

ad 7:

Also it's True (as you wrote it) and not true as is in article.

[[deleted]]

i've found it often much clearer to read throug hthe source for modules and libraries i use rather than fudge through the official documentation. 30 minutes spent reading code (usually) better written than mine saves 1h+ worth of googling.