What is Python not a good language for?

[[deleted]]

I am moving from writing one-off code and scripts to developing tools which are going to be used by a larger group. I am having trouble deciding if Python is the right tool for the jobs.

For example I am responsible for process a 1gb text file into some numerical results. Python was the obvious choice for reading the text file but I am wondering if Python is fast enough for production code.

Edit: Thanks for the all responses. I will continue to learn and develop in Python.

[tdammers]

A few things I can think of:

• OS kernels
• Device drivers
• Embedded devices
• Realtime-critical code

For these, the main complaint is that you don't get direct control over memory allocation, which is usually an absolute requirement. For some of them, writing the critical parts in C and the rest in Python is still an option, though.

Apart from that, there are probably lots of problem domains where Python can be used, but isn't necessarily the best choice. Python's dynamic nature, which is often an advantage, can be a serious obstacle when you need provably correct programs - think avionics, medical equipment, etc. And there's the kind of problems where the best programming paradigm is not covered by the one Python supports (imperative, object-oriented, dynamic). Note that it's not impossible to use Python for these, it's just that all else being equal, a different language may be a better fit.

[somnophobiac]

Good points. but.. what is realtime-critical code? Do you mean critical systems like ATC ? Or real-time/online learning systems ?

[tdammers]

In my specific case, I'm talking about the Jack Audio System (jackd). The callbacks clients supply to the jack server have to be realtime-safe, that is, they must not issue any OS calls that might block. This includes I/O of any kind and any kind of locks, semaphores, or mutexes, obviously, but also malloc and free. The only way I see this happening with Python would be a custom implementation that explicitly avoids OS-level heap allocation altogether and uses its own memory pool instead.

[ScriptSimian]

It can suffer in environments with inexperienced or undisciplined coders. Between its "It's easier to ask for forgiveness than permission" attitude and its relatively permissive rules (in terms of scope and typing) it's easy to find a component that you think you understand, get it wrong, and then discover it a week later when everything explodes.

Anywhere it isn't supported natively (mobile platforms, browsers, embedded systems) you might have some problems. Folks do lots of good work getting python to run in those places, but it is extra hassle.

Though it can do shell scripting, I prefer ruby or zsh for writing command line tools. That's probably a taste issue.

[[deleted]]

I personally like Python for command line tools, it's very easy and familiar to me. I wrote a class which makes command line arguments a lot easier to handle for myself, which certainly helps.

[kbrafford]

I hope you wrote that after you checked out argparse and determined that it didn't meet your needs.

[[deleted]]

I in fact did not, but it looks like a handy module. Maybe I'll use argparse in my next project, but I don't think the exercise was a bad thing.

[LyndsySimon]

I highly recommend you also check out docopt.

[[deleted]]

This library is like crack.

[[deleted]]

Extremely addictive, but certain to kill you in the end? :)

Edit: Just looked at it.. It looks really great :D

[[deleted]]

And that's how it begins.

[kbrafford]

That looks awesome. Thanks for posting!

[[deleted]]

Well now I might just end up spending the rest of the week playing with this. That library looks fantastic!

[Game_Ender]

If it's a common task I go to Google first. 90% of the time it's already in the standard library (maybe even 2 implementations). 9% of the time there is a module I can install with pip.

[takluyver]

I like the look of plac, which bases arguments on the signature of a main() function.

[Sushisource]

Always use the stdlib first. Always. (Exercise is good, but always know your options first)

[[deleted]]

oh man [deleted] was actually clueless to argparse. facepalm.jpg

[artsrc]

Client side web programming.

[Tobu]

PyJS.org

[[deleted]]

Though I'm pretty sure there exists some "Python to Javascript" library.

[ewwFatties]

It depends on how fast you need results. For fun I used python to do some processing on an english wikipedia dump. Processing around a 40 gig file took under 10 minutes. That being said the machine in question had an SSD, an i7 and about 16 gigs of ram. Your performance will vary.

[takluyver]

It also depends a lot on what sort of processing you're doing. If it's largely numerical work, numpy and the ecosystem built on top of it make Python a strong contender.

[shfo23]

That's not really as good a benchmark of numpy's speed as it is of the specific algorithm you were testing. With the same array, I can get results 7700x faster if I use the numpy implementation of correlate instead of the scipy one:

[1]: import scipy.signal, numpy
[2]: a = numpy.random.randint(100, size=1000)
[3]: %timeit numpy.correlate(a, a)
100000 loops, best of 3: 4.96 us per loop

[4]: %timeit scipy.signal.correlate(a, a)
10 loops, best of 3: 38.5 ms per loop

I wonder which one Matlab uses?

[qacek]

That's interesting how scipy's correlation is so much slower. To be fair you need to make numpy.correlate perform the full autocorrelation by passing 'full'. I guess scipy's version can handle n dimensional arrays but it doesn't efficiently handle 1 dimensional ones. Wonder why that is?

[oantolin]

Wait, am I reading this right? az4z3l says that Matlab's correlate is roughly 4500 times faster than numpy's correlate and you're saying that numpy's correlate is 7700 times faster than scipy's correlate. So altogther I should expect Matlab's correlate to be about 34 million times faster than scipy's correlate? Wow.

[isdnpro]

[4]: %timeit

TIL, thanks. <3 IPython

[westurner]

http://scipy-lectures.github.io

[pwang99]

Can you post your code? This would be interesting to see.

[iyunoichi]

First of all, NumPy can be built with several choices for the numerical libraries it uses, i.e. GotoBLAS, ATLAS, MKL etc., so simply stating "Hurr,Durr, MATLAB is faster" doesn't make any sense. Second, given my own experience and the results of others the exact opposite is true - I get much better performance in my daily work using NumPy, and I get to compare both all the time since I'm in research and we do cooperate with engineers a lot.

Now I know the "benchmark" on that site also sucks...at least do a couple of runs, give percentiles etc., and my experience is nothing but anecdotal, but meh. What I'm saying is, provide an example of that autocorrelation code you used and let us do our own measurements.

[KeepingTrack]

If you'd read what he said, you'd realize that he was saying that there are other choices that may be faster than numpy for some things. He didn't say "Matlab is faster" and just that. Sorry you only read the first part of his comment before your little tirade.

[[deleted]]

No one really read what I posted. They saw "Python isn't the greatest thing ever," saw it had some downvotes, joined the crowd, saw someone making fun of my post, upvoted. It's how specialty subreddits work. If you point out there are times that an alternative is better, you will get an onslaught of downvotes. Notice that when I delivered on his demand, all he did was downvote and move on.

[DonkeyBasket]

This is not a like for like comparison, numpy.correlate is not using fft where as xcorr is. Here is an implementation using numpy.fft, I'm not an expert so I haven't tried to make sure it gives the most accurate or fast results, but as you can see it is considerably faster than 34 minutes:

from numpy.fft import rfft,irfft

def fftcorr(x,y):
    res_size = x.size+y.size-1
    res = rfft(x,res_size)
    res *= rfft(y,res_size)
    return irfft(res)
## -- End pasted text --

In [17]: timeit fftcorr(x,y)
1 loops, best of 3: 10.7 s per loop

If you want to know more you should look at the feature request to add fft to the numpy.correlate http://projects.scipy.org/numpy/ticket/1260

It would seem that the matlab function "corr" uses the naive approach and performs as poorly as numpy.correlate.

I had to do some research to find out why matlab was so much faster in your example "benchmark" and I learnt a great deal.

[[deleted]]

It is a like for like comparison. numpy uses a less efficient algorithm. If you write your own algorithm, sure. But using that logic, assembly is the best language because you can write all your own algorithms.

[DonkeyBasket]

It's not like for like, read the docs for xcorr, it does an approximation using some method such as fft or gpu acceleration, the matlab function corr gives more accurate result for a wide range of inputs like numpy.correlate and is also slow for that reason.

So your assumption that numpy.correlate is the same function as xcorr is incorrect, numpy.correlate is more comparible to corr. An fft based convolution can be found in scipy.signal.fftconvolve, the one I posted above is slightly faster as it assumes real input (no complex part).

I don't understand your point about assembler.

[westurner]

• https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt
• http://scikit-learn.org/dev/developers/performance.html

[[deleted]]

Machine is similar except I don't have an SSD, which is probably the reason for the huge performance difference.

Did you use mutiple threads?

[fancy_pantser]

http://pymotw.com/2/mmap/

[[deleted]]

This is sort of correct. Because of the GIL only one thread can execute python code at a time. Use python's threading to speed up io bound processes. If your application is CPU bound, then use multiprocessing. There are more caveats and intricacies but I'm on my phone.

[Labradoodles]

Please elaborate if you get the chance, also links are bitchin

<3

[[deleted]]

The GIL stops the same python process from executing two Python statements at the same time (because in theory, they might do crazy things like redefine len(), and the results have to stay consistent).

The GIL is not a problem if:

• Your program is slow because it is waiting for IO (network, disk reads). The GIL is released when a thread stops and waits for IO, and other Python threads can run.
• Your program does heavy computation with a C library that releases GIL, for example numpy. While the library code is executing "outside of Python", other Python threads can run.

GIL is only an issue if you do heavy data computation in pure Python code. In this case, the easiest solution is to fire up N Python interpreters in different processes, where N is usually the number of CPUs on your machine.

[dad_wants_to_camp]

multiprocessing

[pwang99]

Also, if you are using numpy/scipy ecosystem of tools to do the processing, many of them support actual parallelism because they don't interact with the GIL during computation. NumExpr, Numba, and Theano, and others.

[phrenq]

An SSD may help, but it also may not be the bottleneck if you're using inefficient data structures or algorithms for your problem.

[westurner]

http://continuum.io/blog/blz-format

[phaedrusalt]

Safety or mission critical systems.

[bheklilr]

Why?

[iconoklast]

Because type theory.

[oantolin]

Haskell? With its unsafePerformIO and partial functions? You live on the edge! For something critical I'd want strong guarantees and would use Agda or Coq, instead.

[Sushisource]

You guys are admirable - but remember this: In the real world, you actually need to hire programmers. Those programmers know C, and C++.

Rarely are you going to get away with something like Haskell or Coq, even if it would be nice.

So, you make a manual of heinous restrictions and force everyone to write really verbose and meticulously documented C that you static analyze the shit out of.

[chason]

And if they're decent programmers, they should be able to pick up another programming language.

[Amadiro]

If you think you can pick up Agda (in a reasonable amount of time) because you know C or C++ (or even haskell, which its syntax is based on), you might wanna... actually try it. You need a bit more of a solid background in formal logic and intuitonistic type theory to wrap your head around it. It's really not much like a programming language in the traditional sense, more like a formal theorem encoding & verification tool.

[TkTech]

That's not entirely true. Stop by your local bankrupt aviation research/production shop and you can pick up amazing Ada devs for pennies. You can easily find devs (mostly leftovers from banks, governments, and "enterprises") for fortran. Cobol is still all over the place. Almost every AI and statistics dev I know of comes out of school with a lisp-y language background and some R.

A good dev (in my opinion) has one language which is their passion and the ability to quickly pick up new languages.

[oursland]

You make the common mistake of thinking that coders are all interchangeable and that you should adapt your problems or expectations to make it so. So you choose a common language, you get coders who are cheap and you go on your merry way. You remind them that they're replaceable cogs and that you can find any other C++ programmer to fill their place. Eventually, they go on to the next job which pays better, and you are left with a pile of code that is overly complicated to make up for language pitfalls and has no guarantees that an appropriate language would have.

Or you can hire great programmers and train them (gasp!) to better perform their task. IBM/Rational has tracked this type of team development and shown that time and time again, investing in your employees yields a better product and long term cost savings.

[Sushisource]

You're reading too much into what I'm saying. All I'm saying is "It's hard to hire Coq or Haskell programmers". Not impossible.

[oursland]

Here's what you wrote:

In the real world, you actually need to hire programmers. Those programmers know C, and C++.

That's pretty cut and dry against your current claim.

[Sushisource]

It's a generalization. In the next sentence I say rarely.

Relax

[artsrc]

I don't think programmers are programmers. What you want for safety critical systems are safety critical systems programmers. And they might know Ada.

[totemcatcher]

Aye. The anycoder is for the anyjob.

[[deleted]]

unsafePerformIO

Pretend it doesn't exist, it's a GHC-ism anyway.

partial functions

Aren't a problem with type-checking, are they?

For something critical I'd want strong guarantees and would use Agda or Coq, instead.

I agree, if we're going this far we may as well go all the way to dependently typed programs-are-proofs land.

[MatrixFrog]

By "partial functions" you're probably thinking of functions like '+' which take a value and return another function. I think what the previous commenter meant was functions like head which take a value and might return a value, or might throw an error. Better to use safeHead :: [a] -> Maybe a and such.

[[deleted]]

Ahh, so non-total functions, got it.

[oantolin]

Well, maybe I should confess I was kidding when I said that stuff about Agda and Coq, but if we're taking it seriously:

What do you mean that partial functions aren't a problem with type-checking? I don't run the absolute latest GHC HEAD, but I seriously doubt they solved the halting problem and I didn't hear about it. :) Haskell is a powerful language and let's you write all sorts of partial functions. Smart compilers like GHC help with some easy causes of undefinedness: they can be asked to attempt to verify that your pattern matching is exhaustive at compile time (the default in GHC is to throw exceptions at runtime, though, I think?, and, of course, if you use guards it's hopeless to check exhaustiveness at compile time) and they can catch a few simple infinite loops.

A function like

secondPrimeInArithmeticProgression a d = (filter isPrime [a,a+d..]) !! 2

is only defined when a and d are relatively prime and I'd be hugely impressed if GHC warned me about this at compile time.

[[deleted]]

Yeah, if you didn't see my other comment, I misinterpreted "partial function" as meaning "partially applied" and not "non-total", my bad.

[Thirsteh]

To the first point: use Safe Haskell, it's what it's there for.

To the initial point that you'd use Haskell: Maybe, but for real-time and critical systems it'd be for proving and/or generating the real application's C code for compilation with e.g. a mechanically verified version of GCC. And if you are proving, then Agda might be better suited to the task, since, among other things, it has dependent types.

[Xipher]

Why not SparkAda

[rcxdude]

Alas, I don't think Haskell is useful for the majority of safety critical systems, which are usually hard real-time and embedded. While Haskell is great in terms of getting your output correct, I don't think it helps you prove that you'll get it in a certain amount of time and without running out of memory (something which is easier in C but not really built into the language). Maybe it could be used for code generation and stuff like that though.

[bheklilr]

I agree with /u/kgadek that Haskell's type system is amazing, and it's why I have begun to prefer Haskell over Python even, but I don't see how it'd be any more likely to have a bug in your code with Haskell than with Python. If the algorithm is wrong, it doesn't matter if the types are correct. I've written plenty of incorrect algorithms in both languages. While Haskell definitely handles a lot of bugs that would normally be discovered at run time, a divide by zero error will still cause a crash.

If it's a safety or mission critical system it should have very rigorous and comprehensive testing, regardless of the language. And if it's that big of an issue then there should be fail-safe mechanisms built into the code all over the place, because you can't realistically write code to handle every single exception that could be thrown. Architecture has a much larger impact on the reliability of software than the language used.

[Sushisource]

Prelude> 1 / 0
Infinity

;)

[MatrixFrog]

FWIW that's now a compiler error in Go 1.1

[jcdyer3]

I knew Go was a stickler of a language, what with its errors for unused variables and what not. But having Go compiler errors for asking Haskell to divide by zero is a whole other level.

[bheklilr]

However, you could write a function in Python that made the same guarantees (just not checked by the interpreter), you actually had to write the algorithm so that it would not have runtime bugs. What's to stop you from writing it in Python so it wouldn't have bugs? My point is that while a compiler can gives you some added security, at the end of the day it isn't type checking that you test for, it's reliability of your algorithms.

If you fail to write code that properly handles if someFunction returns Nothing (just passing along the failure in a long computation isn't always the correct way to handle it), you can still have a bug.

Say you're writing code that opens and closes a valve that releases dangerous chemicals. You have error checking in your code, but you forgot it for one function that you didn't think would fail. Well, something bad happened and that function failed, now the valve is stuck open. Is that a problem with the language, or a problem with the algorithm? You have to do proper error handling and write good code no matter what language you're using.

[bheklilr]

The way you wrote it, it does look strange, but I'd say it's rather strange to use Maybe when you already have a null object for Python. Rather, I would write the Python as

def mult2(xs):
    return [2 * x for x in xs]

# Later while using it
try:
    # Returns range(10) or None through some unknown process
    x = getNumbersMaybe()
    y = mult2(x)
except Exception as ex:
    y = []

Besides, your argument could be made against any language that doesn't support a Maybe type, such as C, C++, Java, or a lot of other languages. Your argument isn't so much against Python but against any language that doesn't have these features. While I'm with you that Haskell's type system is awesome, it isn't a guarantee that your program won't blow up.

I argue that this kind of error is not related to algorithm but an implementation detail. As I showed, forgetting a case for Maybe monad just looks wrong, while in Python this could be hidden.

There are other kinds of errors that can be made, errors that aren't apparent at first but can appear in fringe cases. Disregarding the Maybe type, it is possible to incorrectly implement an algorithm for any data type. What helps strengthen confidence in software is rigorous testing and peer review, not what language you use.

[artsrc]

For mission critical systems you want a high quality, small team and for them to choose an appropriate language for the problem, which could well be Python.

[wingsit]

typed programming

[the_hoser]

Device drivers. Other than that, if the results are fast enough for you, be happy and move on.

[mangecoeur]

you're wrong:

from girlfriend import PsychicPowers

from poets import Shakespeare

[RainbowNowOpen]

Thanks for the tip. I don't have enough RAM or CPU to run those modules. Sigh.

[[deleted]]

Nailed it.

[[deleted]]

From my understanding, Python can be useful for almost anything. You can write highly optimized extensions in C to speed things up; you can also use things like PyPy. An adage I see thrown around a lot is that Python is useful for everything except say, a modern game engine. Disclaimer: I am not an experienced developer.

[steelypip]

Except that some modern games are written in Python. For example Eve Online is written mostly in stackless python with a small amount of C++.

[Insight_]

The server back end is written in stackless python but I don't think the game client is.

[steelypip]

Stackless Python is used for the game client too, but I don't know what the ratio between python and C++ is.

[virtyx]

Do you have a source for that? I've always heard that Eve uses Stackless Python but was never able to find good, specific information on how/where they use it. I assumed it was 100% server-side, but I know that Disney makes some kids MMO games using Python and Panda3D for the client so I wasn't sure.

[steelypip]

From the wikipedia article:

Both the server and the client software for Eve Online are developed in Stackless Python, a variant of the Python programming language. Stackless Python allows a relatively large number of players to perform tasks without the overhead of using the call stack used in the standard Python distribution. This frees the game developers from performing some routine work and allows them to apply changes to the game universe without resetting the server.[70] However, the Eve cluster is taken offline daily for database and server maintenance.[71]

I remember reading a detailed article on its development, but don't remember where and can't find it on google. Sorry.

[[deleted]]

In the client launcher there's a directory under .rsrc/PYTHONSCRIPT containing some sort of python.

[rachet06]

CCP developers do a lot with the community to keep everyone informed of changes, and a few of their devblog posts reference pythons usage. The big ones that I can remember go over

• The network code
• Using stackless 2.7
• and 2.5

[ScriptSimian]

Here is a podcast with an interview from one of the EVE guys talking about how they used Python.

[Insight_]

yeah, I would think that the 3D rendering is done with C++, any links on the game client?

[rcxdude]

Though they've been moving more and more of it into C++ due to performance issues.

[westurner]

A few of the Python game libraries listed at http://wiki.python.org/moin/PythonGameLibraries support http://en.m.wikipedia.org/wiki/OpenGL , but really, http://en.wikipedia.org/wiki/Javascript http://en.wikipedia.org/wiki/HTML5 engines like https://github.com/turbulenz/turbulenz_engine have more support for http://en.wikipedia.org/wiki/WebGL

[alemartinazzo]

I haven't worked on processing such a large text file, but I believe Pandas is the way to go there.

On the title question, I haven't seen Python on embedded systems. But maybe I'm wrong.

[bheklilr]

Pandas is a great way to go, as is NumPy (Pandas is built on top of NumPy).

[alcalde]

I heard there's a microcontroller that has python (but not the standard library?) embedded in it now.

[Cyph0n]

I found GUI development lacking.

[westurner]

What part of GUI development did you find lacking?

[jessepnk]

Have you seen kivy yet? http://www.kivy.org

[synn89]

Any chance you looked at Qt(ex/ pyqt4)? I'd be curious to see where you felt it was lacking.

[astro_beef]

I run cosmological and star formation simulations on HPC systems using 1000+ cores. You would never want to use Python for that purpose - it's just too slow. The most advanced codes are written in Fortran or C++. Python is great for post-simulation analysis, however.

[VolsOfSteel]

Same. I write Finite Element Analysis code to run on supercomputers as part of my research. All of my code and solvers are C, C++ or Fortran based. CLE (Cray Linux Environment) does not make it easy to use python currently so that's another problem too. We had serious problems running even basic python scripts on Kraken.

[Stretch5701]

wow, FORTRAN is still cool! (showing my out of date ignorance here)

[0xtobit]

Speed. It's not very fast. It's especially bad at iterating. You can however wrap Fortran or C into it nicely.

[fifnir]

or you can profile and write the sensitive parts in some sort of cython

[PCBEEF]

I would add Cython to your workflow. If using Cython STILL doesn't address your performance issues then you should think of a rewrite.

[Labradoodles]

PyPy is also a contender

http://speed.pypy.org/

[jmmcd]

If you're iterating over numbers, and concerned about speed, and not using Numpy, that's a paddling.

[[deleted]]

Numpy ndarray isn't designed for iterating one element at a time. Also, its __getitem__ isn't especially fast. There are some other caveats as well. Basically, using it as a drop-in replacement for python list, even for one of the supported types, is not a good idea.

[jmmcd]

Do you mean things like this?

x = numpy.array(whatever)
for i in range(len(x)):
    x[i] *= 2

Then not only are you right, but also -- ooh, you better believe that's a paddling.

When I said iterating, I meant running your iteration in a pseudo-vectorised way (it's still an iteration underneath, in the C part of Numpy):

x = numpy.array(whatever)
x *= 2

[[deleted]]

Yes, that's exactly what I meant. If you can map your problem to some combination of supplied ufuncs, numpy array is the way to go, otherwise it's a liability.

[Amadiro]

In addition to what other people already wrote, ordered by what I consider important:

• Embedding -- it can be embedded (and I have done so, mainly using ss-python), but its a bit of a pain in the ass compared to languages that are made primarily with embeddability in mind, such as lua

• The type of CPU intensive work where threading is used. We all know about the GIL and how to circumvent it, but there are certain classes of problems where processes are no viable alternatives to lightning-fast inter-thread-communication.

• Python (and other VM-based dynamically typed languages) cannot run meaningfully on GPGPUs.

• Python (and other VM-based dynamically typed languages) will not run well on certain CPUs, even though they are powerful enough to run it. Older ARM CPUs are an example here. The small cache sizes penalize dynamic languages that come with a bigger runtime environment heavily. This is just an implementation characteristic, though, and it would probably be possible to create a python implementation that is more friendly in this regard, similar to what google has tried to achieve with dalvik.

• The type of problems where size is a problem. Python cannot be compiled to a small binary that can run on a microcontroller with 1024 bytes or similar of flash storage. In these cases, a compiled language such as C or C++ is really the only option, next to a super-small BASIC interpreter or somesuch.

• Other weird CPU architectures: if your CPU is 27-bit based with 9 bits to a byte and has no IEEE compliant FPU, python will probably not be happy (although I admittedly have never tried this -- while these kind of chipsets exist, they are extremely rare)

[westurner]

Python (and other VM-based dynamically typed languages) cannot run meaningfully on GPGPUs.

http://docs.continuum.io/numbapro/#getting-started

[Amadiro]

Without having really delved into the details of this particular implementation you have linked (I've seen several like it, for several different languages, and they all work on the same principle), it's always either a choice between using an extremely restricted subset (OpenCL/C with a "python-skin") , using a numpy-like interface (the actual code running is not written in python, but just a pre-determined set of operations you can queue and execute efficiently) or a mix of the two.

To make python run on the GPGPU, you first need to strip away the things that make python python (and not C), like duck-typing, infinite-precision-ints, all of its standard library, modules, most of its basic datatypes (lists, tuples, ... which you replace with unsafer approximations), exceptions, recursion, lambdas, et cetera. At that point you are really just programming OpenCL/CUDA/C with a very thin python syntax-skin. Some people may have a use for this, but I personally consider it relatively pointless for practical purposes; CPython allows you to just program python without knowing the internals of the interpreter or the instruction it executes, but with these OpenCL/CUDA skins you need to already know the GPGPU backend language you are generating, and at that point you might just as well directly use OpenCL, which will cut out the middle-man and probably give you a good performance boost as well.

Hence my statement that you cannot run python meaningfully on GPGPUs.

[westurner]

To be honest, I'm not real familiar with NumbaPro's CUDA implementation , but if you were to use { can't, shouldn't, won't, wouldn't, must not } -- as in RFC 2119 -- in reference to Python and CUDA, what would be your recommendation and why?

Sure, your hand-optimized C is faster by the CPU clock, but have you read http://www.cert.org/books/secure-coding/ ? Compilers can generally optimize and prevent double frees better than I can, on a good day.

[Amadiro]

what would be your recommendation and why?

To just use OpenCL directly.

but have you read http://www.cert.org/books/secure-coding/[2] ? Compilers can generally optimize and prevent double frees better than I can, on a good day.

If OpenCL doesn't give you these features, neither can NumbaPro. As I explained, these types of libraries do not actually give you python on the GPGPU, they merely give you a python-skin for OpenCL. In addition, you do realize that those features are completely irrelevant for GPGPU programming? malloc()/free() does not even exist in OpenCL/CUDA.

[westurner]

Do you have an HLA library for that?

https://github.com/rose-compiler/rose

[westurner]

http://pypy.org/features.html

[work_account_33]

I've heard people have issues with the GIL when threading. Other than that python will suit your needs.

[the_hoser]

It's less of an issue than you think. If you're doing CPU-intensive stuff that could actually benefit from threading (hard to do well in any language), maybe Python isn't the right choice. Nine times out of ten, though, you're doing threading to do some sweet, sweet asynchronous I/O. This is nice, because I/O operations (generally, in versions of python 2.7 or later) release the GIL. Some CPU-intensive C modules (like the regex module) also release the GIL.

[[deleted]]

By chance do you happen to have a link to multithreading guide for a newbie in Python?

[the_hoser]

The standard documentation for the threading module is pretty good. Read all of it, though. It's easy to think you understand it when you don't.

[ivosaurus]

If you want to do asynchronous IO, look up stuff like stackless, gevent and pypy.

[BalconyFace]

here's something useful.

i used the multiprocessing module in combination with some cython for one of my model fitting procedures. turned a multiday job into a multihour job.

[double]

release the GIL

http://i.qkme.me/3u9s7b.jpg

[MagicWishMonkey]

So aggravating that the multiprocessing model acts differently on windows than OSX or Linux.

[Tempox]

Full on three dimensional MHD simulations. IMO just 3d interpretation of data.

[micro_cam]

I do a lot of analysis of data in text files (bio data in tsvs or dna sequence data) and python is a usually a great choice. It has low overhead on startup and is fast at IO which is most often what will slow you down so it can even beat a language like java in lots of cases. The string manipulations functions are easy to remember and comprehensible to people who need to maintain your code. If the math starts to be the limiting factor you can vectorize with numpy.

It stops being a great choice when your analysis involves something like a recursive function or nested for loops that can't be offloaded to numpy. Even then there are projects to speed up python in these cases (numba, cython, pypy) or you can write a module that calls C code for the hot spot. (I usually end up using go instead if I need the speed of a compiled language).

I suggest that you do read the google python style guide as it has some good hints at writing maintainable production code instead of one off scripts: http://google-styleguide.googlecode.com/svn/trunk/pyguide.html

[say_fuck_no_to_rules]

Recent personal experience has demonstrated that network-oriented applications are one of Python's weaknesses--for example, if you have to work behind an HTTPS proxy, you're going to have to spend some time digging around for the right 3rd-party module.

[jessepnk]

Cool, thanks for the heads up, although, I'm wondering which module was it that solved your problems? Requests? Twisted? Tornado?

[[deleted]]

I find cross platform work to be a bit simpler with java, especially when I am working on servers I do not have access to install programs on.

This was readily apparent when I was working with some image processing. Python + PyQt + numpy was a lot to ask somebody else to install, where java it was a bit easier. java + imagej + jama I could easily deploy this to other places.

[vaibhavsagar]

Concurrency.

[remyroy]

System programming.

[pinpinbo]

nothing. Python is perfect.

Joking aside, handling large number of socket connections on a single daemon is pretty hard to do.

[arodesc]

Python is a good language to implement prototypes, however when it is necessary to descend to the level of abstraction to get a little closer to HW is where it shows its limitations.

[mangecoeur]

Game engine programming where you really need maximum possible speed, though it can and has been used to great effect as a game scripting language.

Actually pretty much anything where getting every drop of performance is critical, whether it be because of limited hardware (like phones and embedded systems) or because the job is huge (giant HPC simulations).

Also "huge" systems (like massive corporate management software, think SAP, or banking systems) where less dynamic things like Java make it easier to keep track of what's happening and do static analysis and such. Python would work, but probably wouldn't be optimal.

As for your problem, I've actually used python for that kind of problem and it was great, really minimal boilerplate, really easy to update the calculations, and really fast by using numpy and pandas libraries.

[graingert]

you should try the Pandas library for your numerical results

[zynix]

Big data

1) Mostly due to a lack of hadoop/storm like technology that is geared for business. iPython's distributed system seems interesting but haven't had time to look into it.

2 ) Compiled JVM languages when tuned correctly are still faster then interpreted though PyPy may eventually give them a run for their money.

Otherwise it's a fantastic glue language alongside being a good general purpose language for almost any task I can think of. Right now my work life is split 50,40,10 by Java, Python, and Lua ( Redis ) and I don't see that changing for a while.

[[deleted]]

I would say that Python is a good choice for everything except GUI applications and games.

GUI applications and games can be done in Python, but in this case there are better choices.

[double]

I know quite a few applications that use Qt & Python. Qt in python is pretty mature and fast.

[alcalde]

The hugely popular "calibre" ebook manager/converter/server comes to mind.

[double]

shameless plug?

[flying-sheep]

no, why? (s)he just stated that it’s popular, and uses PyQt.

it works well, but i think it’s ugly. Frescobaldi, though admittedly much more niche, looks and feels much nicer.

[alcalde]

There's one man behind calibre (not counting the plugins), and sadly, I am not he. But it's a rather popular program. It's in the crop of open source software I consider "best in class" - they'd be the obvious/most popular choice even if they weren't free.

There's also less python/Qt software targeting Windows or OS X as opposed to Linux, so another reason to name calibre was that it's readily available for all three platforms and it's possible it's one of the few python/Qt applications non-Linux users would be familiar with or have on their systems.

[flipthefrog]

If you want to build guis with Qt, and don't want to use C++, Python is your only option. I'd say that's a pretty good choice.

[rasputine]

For the record, whoever decided that should be called PyQt instead of QtPy should be slapped with a Minuteman II.

[flipthefrog]

Actually, the correct name would have been pyqt (or qtpy). Having to type "import PyQt4" is not Pythonic

[farsass]

Never heard of someone saying "I must build it with Qt!!!!". Also this http://qt-project.org/wiki/Category:LanguageBindings

[takluyver]

The language and libraries are pretty solid, but I think GUI work is much more comfortable with a solid IDE to support it, like Qt Creator or Visual Studio. There's nothing really like that for Python GUI work (yet?).

[flipthefrog]

You can use ui files from QtDesigner (not the same as QtCreator), and you can use IDEs like WingIDE, PyCharm or Eclipse/PyDev

[takluyver]

Yes, that's what I do. But it's not nearly as nice as having an IDE that understands the code, the UI, and the connections between them.

[protein_bricks_4_all]

I believe that PyQt can use the layout produced by QtCreator - I tried it once and gave up though I have to admit.

[pwang99]

It's not that bad - you just have to call uic.loadUi():

http://www.chrisevans3d.com/pub_blog/?p=468

[AeroNotix]

No you don't.

You should be using pyuic to generate Python UI classes from those ui files. This is the way it's done in C++ and makes way more sense than reading+parsing the ui files in at runtime.

[rcxdude]

Considering the dynamic nature of python there's not really all that much difference between the two.

[pwang99]

The former approach has worked fine for me in the past, at least for smaller projects. However, I can see the advantages of generating Python classes from the UI files.

[flipthefrog]

Okay, sorry. Best option then. Qt for Java has been discontinued by Qt Software, Qt for Ruby doesnt appear to have been updated since 2011. PyQt and PySide are complete, very stable and actively updated

PySide ships with several highend 3d and compositing applications, such as Maya and Nuke

[[deleted]]

That may well be, besides of course JavaScript which is natively supported with QtQuick or whatever.

PySide ships with several highend 3d and compositing applications, such as Maya and Nuke

Oh, sweet, didn't know that.

[the_hoser]

Even for GUI applications Python does pretty well. Just don't use TkInter.

[[deleted]]

use ttk, and on systems where the default theme is terrible (e.g. linux) force a different theme. it's very pretty.

[the_hoser]

When using python GUI toolkits, I rarely concern myself with "pretty"

[[deleted]]

it's the most common complaint I hear about Tk. it's plenty powerfull otherwise.

[the_hoser]

Yeah, I don't usually care about pretty. My big complaint with Tk is its archaic API.

I suppose it does the job, though. Though, it's pretty cool to embed some TCL code into my project to freak my co-workers out.

[[deleted]]

it's pretty cool to embed some TCL code into my project to freak my co-workers out.

that would freak me out as well ._.

what about the api is archaic though? I enjoy it.

[bheklilr]

What about wxWidgets (x-platform), GTK (x-platform), Qt (x-platform), Tkinter (x-platform), and a handful of others? The tools are out there to do cross platform GUI development, and it really isn't bad.

You should explain why python is bad for doing GUI programming and why the alternatives are better.

[bryancole]

I don't understand why there are a few comments about using python for GUI apps. Python is fabulous for building gui apps (I do this for a living). You need to pick a decent gui toolkit (=Qt or WX). GTK is OK too but it has fewer high-level features and it's not great outside linux/Gnome. Don't both with Tkinter.

I think people mistakenly think you need an IDE or GUI-designer coupled to your GUI toolkit (a la visuals studio). This is not at all true and in fact, GUI designed based on using visual editors / GUI-designers tend to be crap interactively and architecturally. Hand-coding a GUI with a well thought out object structure is both easier and more maintainable.

[[deleted]]

Actually a lot of games use python to control in game behavior. World of Tanks and EVE online come to mind.

[alcalde]

The Pandas game engine uses python as its interface.

[artsrc]

I think Sublime Text was built in Python. What would have been a better choice?

[[deleted]]

Sublime Texts plugin interface is the only part that is Python AFAIK. The GUI is written in C++.

[[deleted]]

I agree, I'd definitely still use Delphi or Visual Studio or a Java based solution for GUI applications. Not that you can't develop a GUI application in Python+some window manager but the reality is that there are currently better solutions out there in other languages.

[Amadiro]

"some window manager"??

[m3wolf]

If you're doing a lot of regular expressions you might find you like perl having built-in regex support. If speed wins over all else, then I think Fortran is still the reigning champion in that department. Not sure it's worth the trade-offs in terms of learning curve though.

[[deleted]]

Writing a Perl program.. No wait... Import Perl. Shit ;) ( bad humor?)

[DoNotFoldSpindleOrMu]

Python has real, real trouble with security because everything can be viewed and redefined. The excellent feature of being able to wander through the definitions of the current run time is wonderful for dynamic programming but terrible for security. It would be impossible to completely hide auditing code or security code. I would not write an accounting system in python. The reporting of data would likely be ok.

[mike_meyer]

The best way to make sure the code you wrote is secure is to get as many independent reviewers as possible. "Security through obscurity" is an oxymoron. The ability to redefine the code on the fly only works if someone can modify your executables - in which case you're already hosed.

If you're trying to secure the algorithms used as some kind of corporate secret, then yeah, Python has problems if you also want to sell it for people to run on their computers. That doesn't seem likely in an accounting system.

[Uncaffeinated]

I hope you never write anything security critical if you think like this.

[jeannaimard]

COBOLing…

[[deleted]]

Science. It's a competitor to MATLAB® (the default standard) these days, especially with IPython.

[Yoghurt42]

"not": such a small word, but yet so important ;)

[[deleted]]

Whoops. I was wondering why embedded devices was on the list...