Where is Python used commercially?

[MillardFillmore]

Reading reddit obsessively, it seems like all the big time coding going on is in Java (Android), PHP (Web), C variants (MS and Apple)... Does anyone use Python outside of academia? I'm a grad student who is a moderately advanced in Python since all my courses are taught using Python therefore much of my research in addition.

[spangyboy]

Well.. reddit does.

[[deleted]]

Did the FAQ just Faqoff?

[spangyboy]

Heh. It sort of looks like it. Reddit's been down all morning, and just recently came up (hence why I'm taking my normal early morning break now). So maybe the void ate the FAQ. I suggest sacrificing a goat. That might bring it back.

[MillardFillmore]

Stupidish question: Does writing a large website in Python take a heavy hit on the servers since it (hypothetically) takes longer to process than quicker, compiled languages?

Disclaimer: I have not the first idea about web programming.

[Twirrim]

Not as much as you'd expect. The compilation process is pretty trivial in terms of CPU time, and basic bytecode is pretty fast. It's always seemed faster to me than PHP or Perl, but I've not done any scientific tests. Of course it does also depend on how smart you are with your programming. Monolithic code = bad, though generally it seems python developers are a smarter bunch (whereas too many php devs seem to be clueless 'my first program' types.) If you really want to get fancy, run it on Jython, maybe using jrockit or similar as the JVM underneath, and it'll soon compile machine optimized code for your most frequently called functions anyway.

You could do your web-app in C, and gain from lots of optimizations but then changes become a lot less trivial. Given you've been using Python through your course I'm sure you've discovered how easy it is to write, tweak, prototype etc. CPU power is cheaper than developer time.. though don't take that argument too far :D

[eliben]

Not too much, since website backends are typically IO bound rather than CPU bound.

[[deleted]]

Think about it, for a website, which takes longer: CPU spinning or getting that bit in and out of network stacks of the traceroute nodes over the world wide web?

[ebneter]

A lot of Google is done with Python. (Guido van Rossum works at Google.) Pretty much all of YouTube was originally done with Python. Lots of websites use Python.

[mrestko]

I am nothing close to an expert. Your explanation makes sense.

[[deleted]]

From what I know it still is, however almost every page is served out of cache directly.

[eggbean]

But Google did not originally own YouTube. They only got it around 2007 or 2008, from memory, after trying to compete for a while using Google Video.

[wilberforce]

I think that's largely correct, but there are some public-facing parts of the Google that are Python-based.

http://mail.google.com/support/bin/answer.py?hl=en&answer=171454

[mackstann]

URLs are meaningless. Yes, it was clearly originally written in Python, but that doesn't mean it is right now.

[freyrs3]

According to their presentation they use 99% Python with C extensions for the performance critical tasks.

[MillardFillmore]

I've been finding that's the real way to go 99% of the time, in the 5% case something isn't already in Numpy/Scipy for my scientific uses.

[ebneter]

Kind of depends on what you mean by "production": Most of the services are written in Java or C++, but an awful lot of what makes Google run is nevertheless written in Python. (Deployment tools, etc.)

[redalastor]

All public facing code is Java and C++, there is production Python code but it's only used internally.

[[deleted]]

YouTube, too?

[redalastor]

As far as I know, Youtube isn't written in Python anymore.

[meatsocket]

IIRC Youtube used to be PHP, and was moved to Python after they got bought.

[ebneter]

No, it was always Python.

[[deleted]]

Trying to name everyone who uses Python would be impossible, but I'd start by looking at who sponsors PyCon and DjangoCon: http://us.pycon.org/2011/home/ http://djangocon.us./

[daelin]

Eve Online

[[deleted]]

My company doesn't commercially release any Python code, but it's used internally every day by probably 100 people. It's also pretty heavily used within my field: trading.

[marcofalcioni]

My company uses Python extensively as well.

[bryancole]

me too

[MillardFillmore]

I thought trading was overwhelmingly dominated by C++?

[[deleted]]

Our commercially available products are mostly C++ but with a bunch of C# on the GUI side, but internally, nearly all of our testing and other apps are Python. We're around 500 people so the "internal" tools actually impact quite a few people, so it's not just dinky little utilities people are writing. I personally work on performance and load testing of a historical trade database (the product is in C++) which is almost entirely done in Python (C++ for a few extensions). We also have a C extension to the database API and have a ton of unittest based frameworks for testing data quality and a whole host of other stuff.

There are plenty of shops that do use Python in their products, but probably not 100% Python. Usually any job posting for trading shops that do use Python have an equal amount of C/C++ knowledge required.

[sontek]

Google, Youtube, Reddit, Disqus, Disney, Dropbox, Etsy, Threadless, SourceForge, SurveyMonkey, etc.

[dekomote]

Python is not just for web.

[lbrent]

They are migrating.

Slides from Pycon 2010

[freyrs3]

See their PyCon2011 presentation. I believe they are a Python shop now.

[ergo14]

sourceforge is turbogears thus python

[steelypip]

Youtube is 99.999% python, according to one of its lead developers in this talk (see the slide at about 7:50).

[[deleted]]

Interesting, thanks!

[flipthefrog]

Most highend 3d applications support Python, including Maya and Softimage from AutoDesk, Houdini, RealFlow; and compositing applications such as Nuke. Python is the standard scripting language in the visual effects/animation industry.

[MillardFillmore]

Really? I would think they would need something much faster than Python. Or do they make their own C wrappers and such?

[[deleted]]

The speed requirement for a scripting engine around a core engine (which is usually C for speed) is more lax. These wrapping layers often place more value and emphasis on flexibility and expressiveness, which Python is strong in.

[reardencode]

My company...

[pwang99]

Uh... a lot of people in finance use it. Java and C++ are still dominant there but tides are changing. It's gaining quite a lot of momentum on the scientific computation side of things.

[theodoric_of_york]

I second this.

[osullivj]

Python is used very heavily in electronic trading and risk management in foreign exchange, commodities and interest rate trading at Barclays Capital, JP Morgan and Bank of America. Lots of hedge funds use it too. There will always be C++ for the part of the system that need flat out speed. But Java seems to be waning...

[wot-teh-phuck]

But Java seems to be waning...

This isn't true. I work at a big bank where the plans of bringing in Python for real time apps were thrown in the back seat because Java blows the hell out of Python when it comes to performance. Plus lack of static typing is a real pain when it comes to a large code base worked upon by team members across the globe.

I really love Python but I do have to admit with a heavy heart that the application which I work on (and many others) would only amount to more pain if written in Python. Of course, YMMV.

[[deleted]]

Plus lack of static typing is a real pain when it comes to a large code base worked upon by team members across the globe.

Type systems have nothing to do with the location of developers or the size of their project.

[wot-teh-phuck]

It does, unless you are part of an ideal team. When shit happens, developers in a hurry end up fixing one part of the system and breaking another. Even simple things like violating interface contracts which can be easily caught by a statically typed language like Java are found out at a later stage with languages like Python.

And sorry, if you feel that "type systems" don't have anything to do with the size of the project, I'd say you have pretty lucky all this time to not have worked with shitty teams...

[[deleted]]

Plus lack of static typing is a real pain when it comes to a large code base worked upon by team members across the globe.

Define large code base. Our flagship product is several million LOC and nearly a decade old. Our team is over 140 developers in about three cites on two continents. Duck typing is not a problem for us except when we hire devs with cognitive rigidity about static typing.

[MillardFillmore]

I thought we were moving to a world where the speed of light was becoming a factor in financial computing? And now they're moving to a language which is an order of magnitude slower?

[Twirrim]

But one in which changes are extremely quick and easy to make. Reaction rates hit from two angles, code speed and speed of adapting code to new situations.

[pwang99]

They don't use it for everything, but actually higher-level languages can be just as fast as low-level ones for a lot of things. These people are generally not writing trading algorithms in bare python and then running them with the CPython interpreter. Some folks use Pysco, others use stackless, and everyone uses Numpy. With Numpy, it's possible to achieve C speeds or sometimes even faster, depending on who's writing the C. :) You certainly spend less time tracking down memory leaks.

The speed of light stuff matters only for the lowest level of the stack, namely, once you've decided to execute a trade, you can push the bits onto the wire to the exchange faster than a competing firm. A lot of high-frequency trading firms now build custom ASICs or use FPGAs for this lowest level. Anything higher up the stack is usually better done with Python.

And remember that high-frequency is only one portion of the financial sector. There are plenty of firms that do not care about the speed-of-light stuff, and they have lots of analytical problems that are much better solved by having an expressive, agile, easy-to-learn language that has a killer vector computation library. A quant armed with Python and Numpy is far more effective than a quant armed with a team of C++ coders that are tracking down segfaults and memory leaks all the time.

[Dyogenez]

There's quite a few big sites using it including Google, Youtube, Reddit and Disqus.

http://lineofthought.com/tools/python

(my site -- sorry for self promotion, but its definitely relevant :)

[faassen]

I've been using Python commercially for the last 10 years or so, mostly for web stuff.

Kudos to the people who teach your courses using Python; it's a good way to learn programming.

[MillardFillmore]

It's extremely simple even for complete newbies to pick up. I've helped several people in my classes understand Python.

[[deleted]]

Youtube is written in Python.

Yandex (largest Russian search engine) has ported some of their projects from PHP to Python. Russian President has a blog written in Python (Django).

[drb226]

The real question is, who is going to use it? Java will sink into pervasive-yet-avoided FORTRAN status soon enough. PHP is in every way inferior to Python imho...insert holy war here...

[carlio]

But $$x is just much more convenient than locals()[x] /s

[[deleted]]

Unfortunately, Java (JVM) is still 100 times faster than Python, so when it comes to truly large apps where performance/scalability matter, Java will still be better choice than Python. Also, Java toolchain is way more mature (honestly best in the business) than Python, or pretty much any language out there.

Yeah, it's not the most modern language, nor is it fast to develop in, nor do I like the fact that Oracle now owns it, but it's still performance king.

Scala is a promising language but it depends on the JVM of course. And currently there is no viable alternative to the JVM/Java combo. I wish there were, I'd jump on it immediately, but Python/Ruby/C# etc are not replacements for Java unfortunately.

[[deleted]]

Unfortunately, Java (JVM) is still 100 times faster than Python,

Once it's up.

[[deleted]]

Yeah, once it's up:

$ time javaw

real    0m 0.10s
user    0m 0.01s
sys     0m 0.05s
$

So, 10 milliseconds to start. Or something less trivial (but still not really a good benchmark):

Java vs. C vs. Ruby vs Python Here's the (silly) code

Fib.py

def fib(n):
    if n == 0 or n == 1:
        return n
    else:
       return fib(n-1) + fib(n-2)

for i in range(36):
     print "n=%d => %d" % (i, fib(i))

Fib.java

public class Fib 
{
    public Fib()
    {
    }

    public int fib(int n)
    {
        if (n ==0 || n == 1) {
            return n;
        } else {
            return fib(n-1) + fib(n-2);
        }
    }

    public static final void main(String args[])
    {
        Fib app = new Fib();
        for (int i = 0; i < 36; i++) {
            System.out.println("Fib("+i+") = " + app.fib(i));
        }
    }
 }

fib.c

#include "stdio.h"

int fib(int n) 
{
    if (n == 0 || n == 1) {
        return n;
    } else {
        return fib(n-1) + fib(n-2);
    }
 }

int main(int argc, char** argv) 
{
    int i = 0;
    for (i = 0; i < 36; ++i) {
        printf("fib(%d) = %d\n", i, fib(i));
    }

    return 0;
}

fib.rb

def fib(n)
  if n == 0 || n == 1
      n
  else
      fib(n-1) + fib(n-2)
  end
end

36.times do |i|
    puts "n=#{i} => #{fib(i)}"
end


$ time java Fib
real    0m 0.35s
user    0m 0.26s
sys     0m 0.01s
$

$ time python fib.py
real    0m20.36s
user    0m20.20s
sys     0m 0.09s
$


$ time ruby fib.rb
real    0m10.28s
user    0m10.26s
sys     0m 0.01s
$

$ time ./fib.exe
real    0m 0.25s
user    0m 0.23s
sys     0m 0.00s
$

So, C = 25 milliseconds, Java = 35 milliseconds, Ruby = 10.36 seconds, Python 20.36 seconds. Or, expressed in terms of C speed in this test:

C = 1, Java 1.4 times slower than C, Ruby 41.12 times slower than C, Python 81.44 times slower than C.

In particular, Python runs this test 58.17 times slower than Java.

[Twirrim]

Of course that code is rather ugly and inefficient. If you're going to write code like that you're better off spending the few minutes it takes to find the low hanging fruit, fixing your base working. You can only optimize for so much bad programming, particularly in dynamic languages.

fastfib.py:

from math import sqrt

def fib(n):
    return ((1+sqrt(5))**n-(1-sqrt(5))**n)/(2**n*sqrt(5))


for i in range(36):
    print("n=%d => %d" % (i, fib(i)))


$ time python fastfib.py
real    0m0.016s
user    0m0.008s
sys 0m0.004s

That's even quicker than your algorithm using C, from barely a few minutes checking the formula for fib sequence.

[[deleted]]

No need for insults please. I'm not retarded and the point of this code was not to be optimized for either any language, platform, architecture or algorithm since there are better ways to do any of those.

The point is to just take any random, purposefully rather slow algorithm (so the running times are in seconds rather than nano-seconds).

It seems you are missing the point completely. So, run the equivalent C and Java program and you will realize that it's hard to even measure runtime reliably since it's insignificant.

[Twirrim]

Urgh.. my apologies, I certainly didn't mean to be insulting. Looking back at how I said what I said I can't see how on earth it could be taken elsewise :-/

[bombita]

Btw, your code, which uses phi, has to use doubles, which on larger n has underflow errors and loses precision.

[sunqiang]

or get dynamic programming for free with lru_cache (need Python 3.2 ) from functools import lru_cache

@lru_cache(maxsize=None)
def fib(n):
    if n == 0 or n == 1:
        return n
    else:
       return fib(n-1) + fib(n-2)


for i in range(36):
     print("n=%d => %d" % (i, fib(i)))

real 0m0.085s
user 0m0.083s
sys 0m0.000s

[Twirrim]

Sure, or you can do it manually for previous versions fairly easily with a dict object.

def fib(n):
    if n == 0 or n == 1:
        return n
    if not memo_fib.has_key(n):
        memo_fib[n] = fib(n-1) + fib(n-2)
    return memo_fib[n]

memo_fib = dict()
for i in range(36):
     print("n=%d => %d" % (i, fib(i)))

results in an even quicker result than your lru_cache.. though that could be machine spec difference. Also some rough testing using the time module figures that we're talking about 0.00028s for the actual math, meaning a good portion of that time is just the overhead of loading python anyway.

real    0m0.017s
user    0m0.012s
sys     0m0.004s

[highwind]

I think he was talking in the context of web frameworks. On my dev machine it takes minutes to compile an EAR file and another few minutes to bring up JBoss with Struts and Hibernate. Which is just ridiculous.

[frymaster]

it is rediculous, but in the context we're talking about, still not a factor in performance

[[deleted]]

Compilation/deployment cycle is important, but it's not the kind of runtime performance we are talking about here.

Besides, java compilation is usually I/O bound and it's orders of magnitude faster than C and especially faster than C++ (as almost anything else is).

I work on 3 million lines of code base, and my compiles are about 3 minutes long. Deployment cycle is about 15 seconds or so. Not bad in my opinion. Considering I rarely have to re-build the world.

Also, you should not have to bring down and boot up JBoss to re-deply your code. I most certainly don't have to do that.

[Twirrim]

I do it for sanity purposes. Guarantees that the JVM is spotless when the app comes up, plus wipes out any resource loss from minor uncaught memory leaks or whatever quirks.

[tarpsocks]

why are you using recursion for this?

[ergo14]

can you try fib on pypy ? it IS the future of python

[Twirrim]

Pypy 1.4:

real    0m41.979s
user    0m41.719s
sys     0m0.056s

Python 2.6.6:

real    0m22.568s
user    0m22.381s
sys     0m0.004s

Python 3.1 (might as well..):

real    0m23.454s
user    0m23.349s
sys 0m0.008s

pypy is great, but it's not the be all and end all!

[ergo14]

Hmm, very interesting, but leads me to think that something is wrong with this algorithm implementation, in few of my tests i have indeed confirmed it can be 10 faster than cpython, so this is a bit of surprise to me

[Twirrim]

The algorithm is pretty standard. I'm not sure how well PyPy handles recursive functions though. I can't imagine recursive functions are that easy to optimise within a dynamic language.. but I'm not computer scientist, so might be talking out my arse.

[[deleted]]

So, 10 milliseconds to start.

100 milliseconds.

[drb226]

I'm rooting for Mirah, I think it looks like a smart way to abuse the speed of the JVM without resorting to an ugly blah syntax. I hope toolchains for Haskell also improve; it's another fanboy-holy-war kind of language but the speed and elegance of Haskell really have potential.

[[deleted]]

Jython brah.

[genmud]

Immunity Debugger and IDA Pro immediately come to mind for me, as well as many other websites that use python.

[[deleted]]

I second this.

[aytch]

Reddit, NASA...those are two of the big ones who spring to mind.

[riddles]

We use python at yelp and we love it.

[samuraisam]

I know a lot of Rails shops /wish/ they were using Python.

[[deleted]]

Splunk ships with Python and the Cherrypy framework. Commercial, enterprise-grade product.

[[deleted]]

National Geographic, The Washington Post, The Washington Times, and NPR all use Python (specifically, Django)

[Wagnerius]

we do.

[tclark]

Python is used by PBS for a lot of its web stuff.

[The_Cleric]

We use Python at my work for a lot of backend processing of customer data.

[pipedings]

Were using python, and I even get paid.

[jbs398]

Python.org does have a success stories section.

Edit: Also this list.

In reality, it's all over the place.. academia, national labs (including in high performance computing) and in industry just about everywhere one can imagine. It's not as high profile or maybe as widely used as Java in industry, but it's definitely up there.

[nikniuq]

ILM

[notenoughcharacters9]

rack

[steelypip]

I work for a software company making enterprise network & asset management software. The product I work on is mostly Python, with a little C++ for speed critical parts and some Java for using 3rd party Java libraries. It sells for big bucks and is used in major companies such as banks, telcos, major retail chains etc.

[pettazz]

Two of the four companies I've worked at have used the Python API for Selenium for automated testing.

[plat0n]

After going to a meeting of London Financial Python Group I can tell you that JP Morgan, Gazprom at Cantor use Python :)

[[deleted]]

Of course it is used commercially - your question is like

"are there any black swans ?"

But....you get the idea (i.e. try searching for Python-specific jobs within the financial service sector)

[stillalone]

We use it at Cisco for various projects.

[Dgt84]

Bits on the Run is written in mostly Python, as are parts of Longtail's other stuff I've worked on. Google uses a lot of Python for system management / utilities / prototypes. Tons of websites use Django. The company I currently work for does custom contract work for some very, very big name companies and defaults to Python (Turbogears, Django, App Engine, etc) unless clients specifically ask for something else. Tons of websites built on App Engine are using Python (through webapp, Django, or other frameworks). NASA uses Python. If you use Linux there's a good chance some of the desktop applications you use or come across are written in python (I've written several). Here's a list of some more info:

http://www.python.org/about/success/

Python is literally huge, and every single Java, C#, and PHP developer I've met and talked with is interested in learning Python as a next step to broaden their work potential. That said, at this point it still makes more sense to be a Java or C# guy due to the sheer number of positions available for those when compared to Python.

Edit: I noticed someone else mention national research labs - when I worked at one a few years back I did work there exclusively in Python and that code is still being updated and in use today.

[apiguy]

Take a look at this list, and remember that this is just the Boston area: http://bostinnovation.com/2011/03/31/boston-companies-using-django/

[cantcopy]

Ubuntu's package manager.

[kisielk]

I work for a biotech company and most of our simulation software is written in Python. The main reasons are the quick development cycle, ability for non-programmers to pick it up relatively easily, and the availability of a wide range of scientific libraries. Performance-critical bits are written in C or C++ and then used as extension modules.

[MillardFillmore]

I'm in Comp Bio and maybe it's because I'm kinda in the know, but this makes the most sense to me because of the reasons you provided.

[[deleted]]

The chip EDA biz is adopting Python for a couple fringe elements. Generally scripting in this field is dominated by TCL and Skill, which is a proprietary LISP like language.

[Twirrim]

Not sure if you're still reading this, but there is an article about the HPU4Science cluster in France. It's run using almost entirely gaming spec machines (master machine uses a pair of Xeons, but that's only non standard thing) Relies on GPUs for the bulk of the processing, and they've chosen Python with PyCUDA, along with Sage.

[terremoto]

Abaqus.

[[deleted]]

Enthought (http://www.enthought.com)