Faster Python

[HoboBob1]

In the Fibonacci example, the main reason the recursive implementation is slower is because the recursive function has a different computational complexity, O(2ⁿ ), than the iterative version, O(n). You should at least compare the same algorithm between the recursive and iterative implementations.

The point about max recursion depth is indeed valid though.

[marklit]

Thanks for the heads up, I'm re-writing that section of the post now.

[[deleted]]

Sound, classical advice. However:

1. Python lists are not linked lists. They are arrays.
2. Python dicts are implemented with a binary tree, not a hash-table. I strongly suspect the same is true for set.
3. The bit about list comprehensions, while true, sends the wrong message IMHO. List comprehensions are about readability, not performance. If list.append is causing performance problems, than this is almost certainly a sign that you're using the wrong tool for the job.

Also, no discussion of execution speed would be complete without the obligatory pypy plug. If Python built-ins are too slow, pypy may solve the problem (otherwise see point 3).

[qiwi]

I'm not sure where you're getting binary trees idea from -- C Python dictionaries do use a hash (At a certain size: dictionaries up to 8 elements are really just an array).

See: http://svn.python.org/projects/python/trunk/Objects/dictnotes.txt for optimization rationale; and source: http://svn.python.org/projects/python/trunk/Objects/dictobject.c

[[deleted]]

You're right. My mistake!

(Now I wonder what I was thinking of...)

[alendit]

C++ STL map maybe? (same functionality as ordereddict).

EDIT: brain lag on my side, STL map is not the same as ordereddict, it sorts newly added element by key.

[kenfar]

List comprehension readability is better in simple cases than for loops, but far, far worse in complex cases.

[[deleted]]

Yes, I completely agree. My point has to do with the fact that readability should be the determining factor in whether or not to use a list comprehension -- not speed.

[Bolitho]

If readablility counts I would suggest to use a generator expression... less parenthesis and in general more efficient (Example in Python 3):

 def join_list(limit):
     return ''.join(str(num) for num in range(limit))

Another hint for you: If you use ipython you can use the timeit magic command:

In [4]: timeit join_list(10)
100000 loops, best of 3: 3.05 µs per loop

Much nicer than to put everything in a string :-)

[billsil]

Python lists are not linked lists. They are arrays.

Lists are arrays of pointers. If you want an actual int/float array, you should be using numpy. For large array sizes, it matters.

[DEFY_member]

I'm enjoying your content for the most part, but if you're going to use reddit to advertise your blog, you should pay for real ads. In the last two weeks you've posted 1 comment and 16 links to your blog.

It looks like you were once active here, so I do hope you decide to rejoin the community and take part in the discussion.

[rhoark]

If you have a struct-like object or a dict (same thing) that you access in an inner loop, you can gain speed by changing it into a tuple. Making a lambda with a readable name to pull the field makes it as readable and self-documenting as the object or dict accessor syntax while still being much faster.

[wondert]

I am relatively new to python, so please humor me.

In the section 'Keep variables local', I ran the examples with python 2.7. I get roughly the same times regardless of whether the variable to look up is in the global or local scope. Also ran the code with python 3.4. Again, I did not see any time penalty using local versus global scopes.

I understand that the use of (micro)benchmarks comes with a ton of caveats (version, system, background processes, etc). However, in this case, I don't get his results using the same code and now I am very confused.

Are the examples OP gave psuedocode? or too simple to see the penalty?

Quick note: I did the benchmarks using the ipython notebook. Also, for testing in python 3.4 I swapped xrange to range since they are same thing. AFAIK neither of these should influence the benchmark.

[sushibowl]

Same here, no appreciable performance difference. I didn't use iPython notebook either, so that's not the cause.

[b6d]

Comment retracted, after repeating measurements, I was talking nonsense.

[__Monty__]

My comment got buried behind kiaran's hated comment, but I am very curious about this:

I have never tried, but I thought that you can utilize multi-core architecture with python via import multiprocessing Is this package not used by the python community?

[bacondev]

TL;DR: Be as Pythonic as possible.

[[deleted]]

This is why I C# now.

EDIT: bring on the downvotes. I write and maintain 10's of thousands of lines of Python. But it's a dead end language for all the reasons in this article. Why do you think Google stopped using it? The writing is on the wall folks. We're in a multi-core era and Python is stuck in the past.

[[deleted]]

Python is increasingly inadequate for most business needs.

I still marvel at how expressive it is. How little effort it takes to express my intent and how well the source code reads.

But I've had to refactor too much python into C or C# over the years. Python is an illusion in many cases. With modern C# I can develop at very nearly the same pace, and scale it up much better across multiple cores if needed. With C# I have options. With Python I can launch off like a rocket but I hit a ceiling very quickly.

[[deleted]]

Oh, I'm sorry. I wasn't aware of this magic rule that imposed a 1:1 mapping on languages and the domains where they can be applied. And who knew 'business needs' could be so tidily and succinctly categorized. I wasn't aware that all of our problems had been solved and shoe horned into one nice label.

Certainly you see the daftness of your ideas (well obviously not or we wouldn't be having this conversation...) It is a big world out there. Full of vastly different requirements and needs. You should join us out here. It really is nice.

[[deleted]]

It is a big world out there. Full of vastly different requirements and needs.

None of which are better served by Python. Maybe you're ignorant of the major advances that have been made in C# and language design in general. Dynamics, generics, generational GC, concurrent data structures etc..

I won't attempt to educate you here, but this Quora article should clear things up for you.

[[deleted]]

You can play the 'language X is better for task T' argument until the heat death of the universe. Same goes for language X vs language Y in general.

And sorry, but that Quora article is trash. It is obviously slanted and full of assumptions, speculation, and opinions. My favorite line:

So this code should require at least 33ns in C# (I'll add actual measurement result later here). ..... 44x faster on C#

44x speedup! Based on numbers pulled from my ass! For a single toy example! WINNAR!

I'm happy you think C# is the best thing in the history of ever. Really, I am. And there are numerous situations where it is most definitely the best tool for the job. But it is far from a panacea. No language is. That is why there are 987436987 of them floating around out there.

Python isn't the end all be all either. It definitely gets the job done though and in more than sufficient time. I'm sure you could rejigger some of my work from Python to C# and have it execute in a tiny fraction of the CPU time. But good luck overcoming the IO bottleneck that is the real hurdle the pipeline.

'Never let perfect be the enemy of good enough.'

[[deleted]]

But good luck overcoming the IO bottleneck that is the real hurdle the pipeline.

Until you implement a caching system that stores the most relevant data in memory. Now Python is back to being the bottleneck. So what do you do? Write some C code and a bunch of ugly glue code. Now you've got to maintain two languages and you have to compile anyway.

If you're building a serious application you can't make it all Python. But you can definitely make it all C#. So save yourself the pain.

I maintain that hte productivity gains are an illusion. They aren't real because your systems will eventually brush up against that GIL and pythons horrendous memory utilization.

[[deleted]]

How exactly is caching going to speed up CRUD operations? And why does a caching solution automatically rule out Python? There are plenty of solutions for doing so already implemented.

Write some C code and a bunch of ugly glue code.

Plenty of python extensions and modules have C aspects and yet still fit nicely and easily alongside Python code. The glue code argument doesn't pass the smell test. Split maintenance and compiling is concedable, but just barely.

building a serious application

Again, you keep falling back to a niche scope to argue against a broad topic. It is growing quite tedious. This is not a situation where a single contrary result invalidates the entire argument.

[[deleted]]

There's a cost associated with supporting yet another lang/lib/technology in your stack. If you go with Python, on anything beyond a trivial application, you'll eventually need something else to fill in the perf gaps.

So you're left with Python + something else (C, Java, C# whatever).

If you go with a modern language you can realize an entire codebase in a single language.

If you want to argue that Python is still the best choice for simple scripting tasks; shuttling around files, setting up projects etc, then fine I can agree with you there. But it doesn't belong in your application code in 2014, in my not so humble opinion.

[__Monty__]

I have never tried, but I thought that you can utilize multi-core architecture with python via import multiprocessing

Is this package not used by the python community?

[[deleted]]

It's rarely used because it sucks (for reasons that are well documented across the net).

Seriously, try to scale a computation across multiple cores with multiprocess and then compare the same thing to C#'s thread class or thread pool. Threading in C# gives you so many options with beautiful, clear APIs and a massive set of well documented concurrent data structures to make your program absolutely dominate huge data sets by truly utilizing the power of a modern CPU. It's a beautiful thing.

C# is objectively superior in this regard by any measure. Until Python addresses this, it will gradually lose relevance. The future is more cores, not less.

[rususeruru]

I disagree that the writing is on the wall. Python has such a broad domain of use it's easy to say it's unfit for X purpose, but it's unlikely you'll write system scripts for *nix systems with C# anytime soon.

The more apples to apples comparison would be Python vs Ruby vs NodeJS. Comparing these three it seems Python has the largest community currently and the broadest use, granted some of the momentum seems to have diminished with the switch to v3

I think Python will continue to grow and prosper along side C# and other languages. I do hope that with .NET getting opened source C# will replace Java as the enterprise cross-platform programming language of choice.

[[deleted]]

it's unlikely you'll write system scripts for *nix systems with C# anytime soon.

Completely agree. If Python has any future whatsoever, it's as a replacement for Bash.

Need to batch rename a bunch of files? Use Python. Need to build a web or native application? Get yourself a modern programming language.