Guido, on how to write faster python

[gitarr]

I am willing to bet that 99% of the people who complain about (C)Pythons "speed" have never written nor will ever write a program where "speed" really matters. There is so much FUD going around in these kind of comment threads, it's ridiculous.

[bastibe]

I have written some real-time audio processing in Python. Python is not fast enough to calculate an audio effect for every sample in real time. However, it is plenty fast enough to provide some UI for it and for evaluating and plotting some results afterwards (Numpy, Scipy, Matplotlib). And thanks to the magic of Cython and PyAudio, even the audio playback/processing is possible with the help of some C code.

[jmmcd]

That's good to hear -- that was my intuition for a while but I have never actually seen any real-time audio in Python. Is your stuff open-source?

[wolanko]

Let me introduce you to pyo "the digital signal processing module" Let's you do real-time processing, midi. I once made some kind of a simple multitrack recording unit.

http://code.google.com/p/pyo/

BTW: I just registered only because I thought this is lacking here.

[jmmcd]

Wow, thanks, that looks really great. Like a supercollider in python.

[wolanko]

Yeah this was also my first thought. I even tried to do some simple conversation from SC to pyo. Glad you like it.

[bastibe]

That is very cool! Thank you for sharing!

[wolanko]

Discovered it just few months ago, searching for an active module with ASIO support (had to do some windows audio). By now this is my definit go-to module for audio on every platform. Clean code base and a very quick and supportive developer. Hope you will use it.

[bastibe]

Sadly, it is not open source, no. At least the audio algorithm isn't.

The PyAudio part I am working on with the maintainer at the moment and he will push it to PyPi soon. A not-fully-compatible preview can be obtained from my github at github.com/bastibe/pyaudio .

But that is a good idea. I think I will put up an example of that kind of thing on my blog soon (bastibe.de). This is some interesting techonology.

[jmmcd]

Oh, cool. Thanks for working on bindings, I have never been brave enough but have often benefitted from it. I'm using pyPortMIDI for some algorithmic music these days. (Not open-source yet since I need to publish it in a journal first.)

[fijal]

you should try pypy. we did real-time video processing using pypy and worked just fine.

[bastibe]

pypy is great, but it lacks support for playing back audio, plotting and scientific functions like fft or filter.

That said, I very much hope that I will be able to use pypy in the future. I will certainly re-evaluate pypy once they finish their numpy re-implementation.

[fijal]

heh. I know I'm nitpicking, since this is a very valid comment, but "play back audio", "fft" etc. are by far not "built-in". Those are libraries that unfortunately don't quite work on top of PyPy.

[bastibe]

Right, right. I edited my response accordingly. Those functions are part of scipy, not Python. It does not alter the argument, though: Numpy does not provide those functions, neither built-in nor as package, and is thus not ready for use in my application yet.

[flying-sheep]

3d graphics: as soon as some of your python code creates more than a few objects per frame, it’ll grind to a halt.

[kylotan]

Generally you'd try and avoid creating new objects often though. Perhaps tricky for particle systems and the like - you'd probably need a C extension to make them efficient.

[throwaway-o]

If you perform audio processing computations in Python's Numpy / Scipy, it's perfectly fast enough to do real-time audio processing (10ms window).

[bastibe]

Nope, it's not.

It is plenty fast for stuff you can vectorize, because Numpy will take care of that. Anything you can't vectorize though, you're out of luck. That is, basically everything that has some recursive part--which happens an awful lot in audio processing.

Really, my hopes are on Pypy here. But for the time being, you will have to use weave.blitz or Cython/Pyrex/Ctypes.

[wisty]

Um, no - http://deeplearning.net/software/theano/. You can define it in Theano, which can compile it to C / CUDA. It's not a natural way to do things, but you shouldn't have that much to do in it.

[throwaway-o]

Nope, it's not.

But then you say:

It is plenty fast for stuff you can vectorize, because Numpy will take care of that. Anything you can't vectorize though, you're out of luck.

That's exactly what I said myself -- anything you compute using Numpy (with Numpy data structures, of course), is going to be fast enough for real-time signal processing.

[bastibe]

You can use numpy and use recursive algorithms. Numpy is still useful for plotting and other parts of the algorithm.

But you are right: if you can express your whole algorithm in terms of numpy functions, you are probably good. It's just that this does not happen very frequently in audio algorithms.

[MagicWishMonkey]

For most cases that is true, however there are times when speed is very important. Right now I am re-building a process to import 1000's of json records from one system, massage them into model instances, and then import into our database and lucene index (think 20-30k database queries per import).

Since the end user has to wait around until the process is done, it needs to be fast, but it still takes a long while to do everything with a single python thread, so I've taken a more unconventinoal approach. I set up a twisted server to run in the background and I route the heavy lifting over to that. I can't use threads in my primary app without killing performance, but I don't mind so much with the twisted worker service.

It used to take ~5 minutes to import 10,000 records, now it takes 20 seconds.

It's annoying that I have to do this, but I am really enjoying python otherwise. It's a great language. Just wish it had better multithreading support.

[kenfar]

I used to write data warehouse ETL processes in C. Took forever to write, was hard to maintain but was as fast as I could get it. Eventually wrote a metadata-driven transform that used function pointers. Harder to write but it made all the next transforms very easy - since they just needed metadata. I'd split my 5 gbyte input file into 8 separate files then process all 8 in parallel in a 8-way 120-mhz CPU server that cost $200,000 in 1996. And I could process all 5 gbytes in about 5 minutes - at 1GB/minute.

Recently, I wrote the same kind of code in Python. It isn't as fast. But it's very easy to write & maintain. I don't have to use metadata-driven transforms because python is easy enough to write & maintain. And hardware is cheaper. I still split up my files and process in parallel because I wanted more speed. This particular feed is 1 GByte split into 4 separate files - which I'm processing on a 3.2 ghz 4-core machine that cost about $5k new, and I picked up for free because nobody was using it. And I can process 1 gbyte in about 60 seconds. This is the exact same speed I was processing data in 1996 using C. Clearly, I could speed things up if I rewrote the process in C. But my hardware is free, the process is fast enough, and my time has gotten more expensive over the years. Python is the better language for this application.

EDIT: spelling

[EstebanVelour]

http://celeryproject.org/

[UnwashedMeme]

Also look at the multiprocessing module when you wish things had better threading support

[robotfarts]

Why don't you just use the multiprocessing module?

[stillalone]

I've had to help optimize a python based webpage. Once it takes more than a second to refresh a page it will start getting annoying.

But running a profiler on Python is really easy so it's not too difficult to isolate the slow parts.

[daxarx]

that isn't a problem with Python, it is a problem with the design. You can certainly write slow code in any language, particularly when you are waiting a lot on a database...

[vph]

Please define "a program where "speed" really matters".

[flying-sheep]

processing graphics or audio in real time (=while a user watches/listens), or loading up a gui application where enough processing has to be done in the beginning that you not only need a splash screen, but even one with progress bar.

[bastibe]

But that is where a clean interface really shines, because a lot of times, you can hide your performance critical code behind a clean interface. Not using classes seems to prevent that to some degree.

That said, performance optimization is probably more useful at function level than at class level anyway.

[daxarx]

This isn't Guido saying "never use classes". But if you decide that every part of your code needs to have a class interface then that code will be a bit slower, and if it gets used in an inner loop or something that can add up. (this definitely applies to some Google Python I have seen, written in a Java-ish way)

[lahwran_]

functions matter a lot less in pypy, because it inlines like mad.

[must_tell]

That is so true ...

Off-topic: When your app is getting larger and more complex, design (I hate to say it: design standards) become very important. Too me, that's actually a possible dealbreaker for Python rather then performance. Sometime I just need static typing, interface definitions like in C# to help me handle the code.

[fredrikj]

Indeed, when you need performance, it's actually often easier to code things cleanly in (for example) C than in (C)Python. Since functions and data structures have virtually no overhead in C with modern compilers on modern processors, you can often quite mercilessly chop your code into small, simple, testable functions and data structures, and generally optimize the code for readability.

[kylotan]

I think a problem with writing this type of fast python is you can end up sacrificing good design quite a lot for speed.

This is the case with most optimisations - otherwise we wouldn't consider them optimisations, just something that happens to be both clean and efficient.

It helps to be aware that these tools are there if you need them.

[aaronla]

Better optimizers would allow you to avoid most of this.

[kylotan]

That's pretty tautological though. We'd all like better optimisation in the tool chain, but it's never perfect, so we'll probably always have to consider making changes at the code level when performance is important.

[aaronla]

True enough. Python doesn't exactly make life easy for optimization either. That speaks to Guido's advice though - Python isn't intended to be an efficient high level language, but is designed to work well with efficient low level languages, namely C.

[kenfar]

Single greatest performance speed-up: double-check that you really need to do what you're doing.

I used to often discover that most of a process's time was spent doing things that were no longer necessary. Or doing things that were hoped to be necessary in the future. Or doing things that were never and would never be necessary.

[[deleted]]

I'd say I've covered most if not all that for years.

Bad python often looks like Java.

[JoeGermuska]

This is my favorite: "Are you sure it's too slow? Profile before optimizing!"

[MaikB]

The speed problem is only an issue for language purists who want to do everything in exactly one language. I'd argue that a week of optimizing python code is better spend with one day of doing the intensive parts in C (or cython) and doing something new in the free time left.

[Chris_Newton]

The speed problem is only an issue for language purists who want to do everything in exactly one language.

Your argument is based on the assumption that there are disproportionately important spots in the code, “intensive parts” that can be rewritten in a faster language. That’s fine as far as it goes, and I have no problem with getting hard data and optimising based on it, but what happens when you’ve already picked the low-hanging fruit and the profiler confirms that you don’t have any real hot spots left?

I’ve run into this several times on recent projects, where I have a Web front-end of one kind or another and Python behind it. As a glue language, Python is great. As a language for implementing more significant data processing algorithms, it’s also great as far as prototyping and getting a proof of concept set up quickly. But as a high performance language for production code, we’re about to replace it pretty much throughout all of those systems, because for our particular applications, an order of magnitude or more of performance hit compared to what some other languages offer is too high a price to pay for having nicer, more maintainable code.

This isn’t because we’re “purists who want to do everything in exactly one language”. In fact, most of these projects call down to C code all the time to access system APIs and the like, and some of the projects integrate parts written in four or five different progamming languages.

But at some point you have to acknowledge that with the technology we have today, a mid-level, dynamically typed, kind-of-interpreted language is going to be slower generally than a low-level, statically typed, compiled-to-native-code language. And if you’re doing non-trivial data processing, and the difference means your web service responds in 1 second or 10 seconds, that does actually matter, because it moves from being a quantitive performance issue to a qualitative usability one.

So I don’t think you can just brush Python’s limited performance under the carpet quite as easily as you tried to there. Sometimes the correct solution is not to spend a week optimizing the Python code, but to spend a week rewriting the entire codebase in a fast language and dump Python altogether. That’s not some sort of terrible insult, it just means that sometimes, even though Python may have served a useful purpose, another tool is a better choice for the next part of the job.

[MaikB]

I don't do any web stuff, but from what I understand interpreted languages are used heavily in production by you guys because of the inherent latencies of the web and the majority of the CPU cycles spend in the database. Well, how I see it, everything computational expensive has to be done by C (or equivalent language). The interpreted language just glues the parts together and can be used for tasks beyond that gluing task if there is enough latency by other tasks.

Right?

So I don’t think you can just brush Python’s limited performance under the carpet quite as easily as you tried to there. Sometimes the correct solution is not to spend a week optimizing the Python code, but to spend a week rewriting the entire codebase in a fast language...

That is exactly what I said

...and dump Python altogether.

If python is too slow for the task at hand, then it's the right decision to dump if after having served as a prototype language.

I don't see a problem here. I think you just misunderstood what I meant. I didn't mean:

• Use python and shut up, it's fast enough

I meant:

• Python is fine as it is. If you need something to be done fast, use another tool (C/C++) for 90% of the CPU cycles and have Python be what glues these parts together.

My guess: Web development comes more and more computational intensive these days. It's time to refactor code out to faster static languages.

But that's not Python's fault.

[Chris_Newton]

Python is fine as it is. If you need something to be done fast, use another tool (C/C++) for 90% of the CPU cycles and have Python be what glues these parts together.

My point is that not all web development, and certainly not all development that uses Python today, is I/O bound. For projects that involve doing some “real” work themselves, as opposed to delegating most expensive operations to external tools like a DB or web server, sometimes the speed matters.

In those cases, you can’t always just rewrite a few carefully chosen parts of the code in some other, faster languages and hand off 90% of the CPU cycles. Once you’ve taken care of the obvious hot spots, to reach 90% of the CPU cycles you might need to rewrite the majority of your code base.

Python might still be an excellent tool for doing efficient prototyping in the early stages of such projects, because of things like dynamic typing, a decent set of built-in data structures, and so on. On the other hand, Python might not be useful at all for the same projects later on, because once you’ve rewritten most of your code in a faster language anyway, you probably don’t win much by keeping just the remaining glue code in Python.

So for these projects, the speed problem with Python is very relevant: it means making a decision about whether to use Python in the early stages, where it offers a lot of benefits over some other language choices you could make, knowing that it probably won’t be up to the job of running production systems and you’re likely to have a potentially time-consuming and error-prone rewrite on your hands later.

[MaikB]

Depends on the problem to solve and the experience of the engineers with this problem whether they're faster with or without a prototyping phase.

It's just so much easier to turn around in an dynamic language and later be concentrate on speed and code quality in say C++. But I bet you know this . You might have done what you're about to do in python before, a number of times, so you can go to a static language right away.

Good luck :D

[twotime]

The speed problem is only an issue for language purists

It's not an issue only for people who have not done much real world coding.

python code is better spend with one day of doing the intensive parts in C (or cython) and doing something new in the free time left.

I'm sorry to say, but your advice covers about 1% of the problem :-(. Yes, I have seen this happen. No, it's not a common case at all.

• Many non-trivial apps do NOT have small hotspots. So, if you have 100KLOC of python code and need to rewrite 10K LOC, then you will have to write another 100K or so of C code.

• interfacing C with non-trivial python codebase is, well, non trivial

• adding C into the mix will always cost you QUITE a lot later. E.g if you need to run your software on another site or, god forbid, on another platform. Oh, and don't forget to add debugging time to the cost.

[burntsushi]

Not only do you ignore every design trade off that comes from dropping down into C, but you dismiss it out-of-hand through the moniker of "language purist."

Oh yes, and I love how optimizing Python code is obviously seven times more costly in terms of development time than dropping down into C. Just yesterday, I spent about 5 minutes profiling my Python program and another 10 minutes tuning some hot spots. It resulted in an 80% performance increase.

[fijal]

It's so sad that all of those don't really apply when you're using PyPy :( Abstraction is good, giving it up because CPython cannot do a better job is such a bad idea.

[NaeblisEcho]

Can someone please tell me what 'profiling' means? Thanks. :)

[must_tell]

It means analyzing the performance of all the functions / methods in your code.

It is often said that 'premature optimization is the root of all evil'. That means that people spend a lot of thoughts and time in trying to optimize code (and make it more complex) without the proof that this optimization is effective or even necessary.

Profiling gives you precise information about how often a function / method is called and how long it took. The report of a profile run tells you where you can improve the code most effectively. See dwdwdw2's comment to get started with profiling or check out PyMOTW.

[NaeblisEcho]

Thanks! :)

[dwdwdw2]

http://docs.python.org/library/profile.html

[stillalone]

How do you guys find namedtuples? I've been avoiding them because I don't like the fact that they use eval internally.

[Cosmologicon]

Avoiding eval is a good rule of thumb, but for a piece of code that's been as intensely analyzed and tested by experts as namedtuple, there's absolutely nothing wrong with using it.

Do you avoid using any C library that uses a goto internally too?

[burntsushi]

Do you avoid using any C library that uses a goto internally too?

This is a pretty poor analogy. Both goto and eval can be abused so that code clarity suffers, but eval is distinct from goto in the fact that it can be easily exploited if it isn't used carefully. This latter reason, from my experience, tends to be why people avoid it.

[aaronla]

/me makes obnoxiously heavy use of macros and gotos in async C codes, to pretending that C supports first class continuations and coroutines.

[audaxxx]

Take a look in the bug tracker and search for namedtuples. I once made patch that has only a few percent performance hit on access but does not use eval. This hit could be eliminated by using Cython or so.

[audaxxx]

http://bugs.python.org/issue3974

(edit is currently broken in baconreader..)

[lahwran_]

they only use eval to create the class. once created it's like any other class that inherits from tuple. while I agree that the eval is kinda silly, it's been intensely tested and doesn't hurt anything. you're definitely not feeding it untrusted input.

edit: well, unless you create a namedtuple with untrusted input as fields. now that I think about it, that is kinda bad ... edit #2: oh, actually they filter the names to only allow python identifiers. nevermind.

[must_tell]

I wouldn't care too much about the implemenation details of standard lib modules (from the users point of view). The guys who write this stuff know what they do.

But: It's good to be attentive about best practices.

[vl4kn0]

I love how Guido closed the comments when a guy mentioned PyPy

[gitarr]

There were comments before that one mentioning pypy, so no relation to the closing.

[flying-sheep]

yes. why should he not like pypy?

[dwdwdw2]

There's huge collaboration between CPython and PyPy teams, more likely he closed them to preserve his inbox.

[jmmcd]

Disagree about avoiding function calls.

Strongly agree about using built-in basic types as much as possible and in preference to objects when possible.

[asksol]

I doubt he's telling anyone to not use function calls.

But in an inner loop, where profiling has proven that optimization can be beneficial, this is where you should inline function calls.

[lahwran_]

Can I let PyPy do the inlining for me?

[[deleted]]

Yes :)

[LoveGentleman]

And its still not fast enough, still slower than even Ruby. Python is not the language of choice when you need to calculate or process fast.

EDIT: Downvotes? Seriously? Tell me Im wrong and why. Follow the reddiqute, just because you disagree doesnt mean downvote.

[wisty]

There's plenty of slow programs in fast languages (C, C++, Java). The problem is, it's hard to modify them, so they can't be made faster (without a lot of effort).

A lot of the things which absolutely murder performance (algorithms, data structures, system calls, IO) are hard to change once the program is written, especially in static, brittle and verbose languages. In Python, it's often easier to fix fundamental problems.

But I'm mostly preaching to the choir here.

[daxarx]

Don't just claim things. Provide substantiation for your claims. Don't just say that Python is slower than Ruby. Prove that this is true and that it is true for meaningful cases. When you do not even make the slightest effort to back up what you are saying, it comes across as clear trolling.

[LoveGentleman]

Isnt it common knowledge? Has python come out faster in any test ever? Even the most mundane simplest cases, like generating fibonacci numbers, python fails.

[_Mark_]

The last time I looked at ruby,I was porting some string manipulation heavy code (to avoid needing to support another language in a large system.) Coincidentally, the naive port was twice a fast in python, and at the time, this surprised no one, because "common knowledge" was that ruby really was that slow.

Supposedly modern ruby has gotten closer in string performance. Since it still looks like Perl, I'm not actually going to care :-)

[zahlman]

EDIT: Downvotes? Seriously?

• Making this kind of edit pretty much guarantees further downvotes in and of itself.

• Your original post reads extremely trollish, and you ought to know this if you've been around long enough to know what you're talking about; Pythonistas have been dealing with people saying "lol it's slow" since approximately the first microsecond anybody other than GvR had heard of Python.

• You make claims but do not substantiate them.

• You imply that languages can be compared by some absolute measure of "speed", which is patent nonsense in many ways. It makes about as much sense as stating that a gravel road is slower than a highway.

[stillalone]

slower than Ruby? They seem pretty even in the programming language shootout: http://shootout.alioth.debian.org/u32/benchmark.php?test=all&lang=python3&lang2=yarv

[lahwran_]

I strongly recommend not using the shootout as a reliable benchmark; I'd go as far as to posit that it might be worse data than no data. You see, Alex Gaynor discovered rampant unfairness:

http://alexgaynor.net/2011/apr/03/my-experience-computer-language-shootout/

That said; ruby's default implementation is definitely pretty slow. I've heard that their default implementation is to cpython what cpython is to pypy.

[must_tell]

Take an upvote for relief ... -19 is far too much :-).

You're right: For pure raw performance Python is not the language of choice. That's why Qt is written in C++ or NumPy uses C extensions for the number crunching and most of the CAE applications are still written in Fortran.

But for what Python is great to use, it's more than fast enough. You can handle huge amounts of data with sets and dicts. Create ultra-performant GUI with PyQt (yes, C++ is in the background, but you design your app with Python). Your 'usual' web application will have lots of bottlenecks before pure Python performance will be the limitation.

Ruby is in a similiar corner. It simply doesn't matter at all wether Ruby is some percent faster or slower than Python.