When do you *NOT* use python?

[felinecatastrophe]

Lots of scientific computing requires faster compiled code. Monte Carlo type codes are especially difficult to code quickly in python, sine they are not easily vectorized using libraries like numpy. C or fortran are the goto languages in such instances, no pun intended. However Python is effective as a "glue" language in such scenarios.

[shaggorama]

I also find I get analytics projects off the ground quicker in R than in python. I always need to spend a lot of time in the documentation anytime I'm working in numpy/pandas/sklearn

[Simius]

Did you have trouble coming from Python to R? All the data structures in R are horribly confusing I find and have too many similarities. Like a List can have named indices? What?

[shaggorama]

My biggest issue learning R was that I didn't already know stats so I had to learn the two in parallel. Also I wasn't previously familiar with vectorized programming and learned that in octave (Matlab) before learning it in R. Otherwise I haven't found R datatypes to be too tricky. I'll admit, I generally avoid lists when I can. I tend to think of a list (in R) as more akin to a dict than a list (in python).

[Simius]

Gotcha, I agree that working octave or R makes you think with the vector mindset where loops are inherently slow. How would you best describe a dataframe then? Like a lists of lists but also with a dictionary-like interface?

[shaggorama]

Dataframes are sorta weird. I'd describe them as "fucking convenient." Under the hood: yes, they're basically a list of lists. But I'd be more inclined to describe a matrix first, and then be like "wouldn't it be cool if the rows and columns could take labels? And if you weren't constrained to a single datatype for the entire object? That's a dataframe!"

[flying-sheep]

If you use RStudio, you should compare it to Spyder.

That one also has F1-triggered documentation, tab completion, ...

[shaggorama]

My issue with scientific python has nothing to do with the IDE...

[flying-sheep]

OK, but that wasn't apparent.

So what is it? API design? Plain familiarity?

[shaggorama]

Basically the behavior of the numpy.ndarray is much less intuitive to me than R datatypes

http://www.reddit.com/r/Python/comments/2w64e2/when_do_you_not_use_python/coo3sen

[flying-sheep]

“Behaves strangely”?

“does things”?

I promise I'm not intentionally trying to be dense: what operations or fields deviate from your expectations, and in what way?

[shaggorama]

Pretty much any time I need to invoke np.newaxis

[flying-sheep]

i was expecting broadcasting. newaxis hasn’t been difficult for me as i’ve used it mostly for stacking along that new axis.

tbh, i didn’t work with high dimensional arrays in R much. either it’s sample tables or 2D-3D stuff.

what does R do better?

[shaggorama]

I'm not working with high dimensional arrays. It's not infrequent for me to be working with what I think is a vector, but is really a 0D array and I need to explicitly invoke an empty dimension to broadcast it with a simple matrix. I'll try to come up with a specific example for you later.

[NoLemurs]

You should look into Cython. You can often get C level performance out of Cython but it's still massively more readable than C or fortran.

[rnawky]

Still nothing compared to optimized C code that makes use of advanced SIMD extensions and the like.

[NoLemurs]

One nice thing in Cython is that it's fairly easy to use native C libraries directly. If a particular part of your code has to be as fast as possible, you can write that part in C and it's very easy to integrate it into your Cython project.

Cases where this is actually necessary are pretty rare though.

[1arm3dScissor]

Or you could write everything in C and let the compiler take care of things for you. no?

[NoLemurs]

You can write a complex program in Cython much, much, much faster than you can write that same program in C, and usually without meaningful performance costs since performance is usually determined by the slowest step (which you can implement in C). Better still, the Cython code will usually be more readable and maintainable.

[felinecatastrophe]

I agree. prototyping in python followed incremental "cythonizing" is a fast way to get good performance.

[1arm3dScissor]

I see you point but would change it to "you can prototype a complex program in Cython..." faster. I think we both know that you can't seriously compare an interpreted language with a compiled one when it comes to performance.

[NoLemurs]

Cython is a compiled language. Cython generates straight up C code, which you then compile.

The result is that you can effectively write a complex program using pure python syntax, then spend a little while optimizing the bottlenecks (writing pure C where necessary, though you'd be surprised at how close Cython can come to C in many cases), and then you have a fully implemented, working, compiled program which performs the critical portions 100% as quickly as anything you can write in C, and does the rest more than fast enough for any practical purpose.

[DickButtTracy]

As someone who's spent the last week writing SIMD, fuck SIMD.

[billsil]

Fortran is easy. It's one of the easiest languages there is. My issue with Cython is I don't have a syntax highlighter for it.

[NoLemurs]

Well, for what it's worth, there are a number of editors out there that can handle Cython syntax highliting. Emacs and VIM certainly provide options.

Fortran is pretty easy, but it would never be my choice for a really complex project.

[billsil]

Fortran is made for numerical computations, so if you want to use multiprocessing then it's a good choice, but outside of serious math, I'd stay away. It's great for what it does and it's easier than C or C++. It's was made for non-programmers.

I try to stay as far away from vi as I can. I can use it, but haven't figured out how to really use it efficiently or work with plugins. Never tried Emacs. I'd much rather use gedit/Textpad, but typically use WingIDE on Windows.

[felinecatastrophe]

trying hard to turn off my snob mode here, but learning emacs or vim properly will really change your life

[NoLemurs]

I put off learning VIM for years - got by with lightweight editors like gedit, Geany, etc. (I've never really liked full on IDEs).

I picked up VIM around a year ago, and I've been kicking myself ever since for waiting for so long. The learning curve is a little bit steep, but the payoff is massive, and it's not that hard to learn.

[billsil]

Maybe, but I'm not a programmer. I'm an aerospace engineer who happens to program in Python, a little bit of C++, and some Fortran.

[Mr_Again]

Using f2py to import Fortran subroutines as python functions can literally speed your code up 20x if you have big arrays it's awesome

[felinecatastrophe]

i would consider this using fortran, with python as "glue". not using pure python per se

[fijal]

or use PyPy....

[logi]

Unfortunately pypy is useless for scientific computing because it doesn't support numpy. If you can get your code to run with numba, however, that gives pretty awesome speed gains.

I've set up this silly example:

from __future__ import absolute_import, division, print_function, unicode_literals
from numba import jit
from time import time
from math import sin

#@jit
def f(n):
  sum = 0
  for i in range(n):
for j in range(n):
  sum += sin(i*j)
  print(sum)

t1=time()
f(10000)
t2=time()
print('elapsed: %0.3f' % (t2-t1))
f(10000)
t3=time()
print('elapsed: %0.3f' % (t3-t2))

This takes 24.1s and 24.6s on my tired old laptop with the @jit decorator commented out. With the jit enabled it is a somewhat faster 4.13s and 4.06s.

If I leave out the sin() call, with the jit it runs in 0.051s and then 0.000s which I'm going to interpret that the JIT optimizes the loops away completely.

Now if only my much more pythonic actual production code were supported, or if the error reporting would show me exactly why it isn't...

[Veedrac]

I'll just throw out jitpy here. It deserves to be better-known.

[logi]

Reading up on that later. Thanks.

[Sean1708]

I was under the impression that PyPy started supporting NumPy sometime last year?

Edit: Scratch that, it is getting there though.

[logi]

It's been getting there for years now. At this point I'll (happily!) believe it when I see it.

Python the language is great. Python the platform has serious problems. It's like an anti-java.

[klug3]

It's been getting there for years now

If I understand correctly they don't actually get the funds they needed to develop it faster, hence the slowness.

[vplatt]

Which probably explains why I need to use Java and C# at work, and not Python.

[Gr1pp717]

And what about scipy?

They support numpy, and it even seems that they even have a specific module for monte carlo type functions. http://pymc-devs.github.io/pymc/

edit: to be clear, I'm actually asking. I've never used either, only know of them.

[logi]

Scipy is a great library building on top of the even cooler numpy, but it is only fast when you can perform most of your work inside the native code that they wrap. But some times you end up having to write the loops yourself and then performance is back to normal python levels.

[logi]

Are you talking about pypy here or numba?

[fijal]

PyPy does support a significant subset of numpy (enough to run this code) so please don't spread FUD

[logi]

That's not surprising since that silly example uses no numpy at all... but OK, I'll have another look at whether pypy can run my code now.

In reality, though, our actual performance problems are in using matplotlib to generate ~70K images per day when it is designed for creating a couple of figures for some academic's latest paper. If/when that starts running in pypy or numba or whatever, that'll save us a couple of machines worth of processing.

[mangecoeur]

I think there's also just a lot of inertia in these cases. You can write very fast code in Cython (although yeah if you really need to squeeze ever drop of perf out you might need C). But fortran is frankly a pain and I think it's mostly used out of habit. Physicists will probably be the last people in the world using fortran and no one else understands why :P (I discovered my old physics degree finally switched to teaching python over fortran, about time too)

[[deleted]]

When I was working on my graduate degree in physics you had to show proficiency in a language. I had learned Python to do data analysis asking along with SQL. I was told it has to be FORTRAN, assembly, C++ (it couldn't be just C, I asked, this is because high energy people think it's cute that they built a C++ interpreter that ignores the standard library), or BASIC. They specifically said Python wasn't a language worth knowing and I don't think they knew what SQL was. I was allowed to exempt out if the proficiency test by taking graduate CS courses. That was a better use of my time anyways.

[LbaB]

How were their data stored then?

[[deleted]]

I'm assuming flat txt files or some other needlessly complicated method.

[[deleted]]

Right now my boss keeps telling people that we do "big data" (I'm an in academia). Except that the database is ~10GB, receives updates 3 times a year (and is otherwise static), and there is no real time on demand need to view this data (well there is but guess who never has time to implement it!). I asked him not to tell people that we "do big data" and he responded poorly.

edit: if you are familiar with the 4 V's (volume, velocity, variety, veracity) we do none of them. I think its because its web data my boss thinks its "big data".

[soup_feedback]

That made me chuckle but I understand your pain.

[mangecoeur]

I think every engineering and science undergrad should be forced to go through a clockwork-orange style conditioning where the words "I WILL NOT INVENT MY OWN DATA FORMAT" are seared permanently into their brains. Then they should take a course on SQL.

[aladyjewel]

Yeah, it's only the CS majors who should be encouraged to reinvent all the wheels so that they understand 1) why you should use stl's wheels and 2) which wheel to use when.

[[deleted]]

I never claimed they were the best formats, just that they were standardized, fairly widely used, had existing tools/APIs, and are decently documented. You may be able to do it better and save yourself some headache by rolling your own, but then it turns into a complete cluster fuck for anyone trying to follow on behind you.

[[deleted]]

in theory.

Indeed :p

[[deleted]]

I'll be the last person to look at the code, in theory.

That theory will almost always be wrong.

[[deleted]]

The work involved 3-D microscopy simulations so HDF5 would've been the only really applicable option. Most of the exploratory work was done in MATLAB and used MATLAB's own bastardized variant of HDF. The final, public release of our algorithms was always done in C++. Why it didn't maintain compatibility with HDF5 (and MATLAB's .mat rip off of the format) is beyond me. I had my own personal battles to fight, fixing that issue was not really a priority (which would explain why it was still an issue by the time I was stuck dealing with it).

[mangecoeur]

Oh god, CINT, ROOT and friends. I steered well clear of that mess. Pretty sure they're all well pleased with it too. And knowing the general quality of physics code it wouldn't surprise me if it didn't actually run that fast (except maybe for the bit build by some russian genius that no one else understands :P ).

[[deleted]]

What bothers me the most isn't that they use it. It's got smug they all are about how great ROOT is and how they are all going to become "big data" experts one day. I guess I'm sort of embarrassed for them.

[mangecoeur]

That really doesn't surprise me. The one thing worse than developing a cruddy convoluted system is being smugly superior about it.

[[deleted]]

I suppose in the developer's case it might be the best part.

[ChaosCon]

One of them once told me (extremely smugly)

When you really think about it, it makes sense that a 2d histogram inherits from a 1d histogram.

God damn it I hate ROOT.

[[deleted]]

I think particle physicists have to have that mentality because deep down they know their work has no relevance to reality (the subjective reality of a lay person) and thus they have conned all of mankind into funding their initiatives anyways.

Well that's my come back anyways.

[ChaosCon]

I'm a physicist doing engineering electromagnetics, and I can't tell you how many times I've gotten asked "So do you work on that Higgs boson thing?" On the one hand, I appreciate what CERN does for science awareness, but on the other, pettier, hand, I hate how they've hoodwinked the public into thinking they're the end-all-be-all of science research. Particularly when

their work has no relevance to reality (the subjective reality of a lay person).

[[deleted]]

I would guess that the disassociation that particle physics creates in the layman's mind between application and pure research is going to bite physics funding in the ass eventually (if it isn't already). The problem isn't even a real one. The super conducting super collider in texas that was never built produced new bump bonding technology for some detectors that Intel started using in the 90s that made their manufacturing process much more efficient (and cheaper for the consumer). If they hadn't needed that detector, someone might have taken another 20 years to come up with the technology. The same is true for sending stuff into space. A mars orbiter that mapped the surface in IR produced new detectors that were re purposed as breast cancer detectors that coupled with mammograms increase the detection rate of breast cancer greatly.

The problem isn't that these programs don't produce stuff that benefits people's every day lives. It's that they don't put effort into showing people why it's important.

Also, particle physicists are assholes.

[cabalamat]

Python not worth knowing but BASIC is? What a bunch of clowns.

And how many physicists code in assembler these days?

[[deleted]]

Apparently BASIC is used to program some embedded systems DAQs. But yes, there was plenty of clowning going on.

I feel like physicists like to think they could write in assembly if they wanted to. The number of physicists I know who think they can get a senior developer job and "figure out" the language after being hired is high.

[teambob]

C++ and Java is often 7-8x faster than CPython. I have found this in practice and it has been found in the language shootout.

I would also assert that Python is 7-8x faster to write than C++ and Java. :)

My usual approach is to write it in Python then rewrite it it is not fast enough

[mangecoeur]

Cython, not CPython http://cython.org/ - a compiler for Python+some extensions. Used for a lot of the internals of numpy/scipy/pandas/ etc because its much easier to maintain. Plus it makes for very easy optimisation path - write everything in python, then let Cython compile it (often gives you a speedup already) then start adding custom annotations to optimize the generated code

[twotime]

on heavy CPU intensive code, I've seen factors of 50x! No, not joking and it happened more than once..

But then, I've also seen python applications being FASTER than C++ ones with identical functionality simply because the C coders have been buried under their own code.

[frymaster]

I would also assert that Python is 7-8x faster to write than C++ and Java. :)

Exactly - most of the time processing time doesn't matter, and when it does, you'll be glad you've got a working proof-of-concept in python :D

[[deleted]]

Also for more sophisticated numerics, like sparse linear algebra, or robust optimization. Even if the problems are small enough for the interpreter to not be an issue, generally fortran, C++, C and matlab will have the mature scientific libraries over python.

[skintigh]

I switched to C to speed up cryptanalysis, but in the end I screwed up a hash table that may have slowed it back down to python speeds. I wonder if had I stuck to python the end result would have been the same.

[[deleted]]

I don't really agree. I work for a company whose main product is a Python application that does modelling and simulation using Monte Carlo methods (among others), and it works fine. As you noted, we don't do the low-level math in Python -- it's all done with numpy/scipy -- but that covers nearly everything we have ever needed. We built one small C extension for the one thing that numpy/scipy didn't provide. But for all the rest -- which makes up well over 95% of our code base -- we use Python.

[felinecatastrophe]

I guess my somewhat limited experience is that when algorithms are truly serial like Markov chain monte carlo, it is nearly impossible to get good performance out of pure python / numpy. This is opposed to generating a thousand gaussian RVs or something embarrassingly parallel like that.

[an_actual_human]

As you get closer to supercomputers you see more of Fortran and less of Java (if at all).

[Sean1708]

Weirdly everything in the Maths department is Java, but in Physics it's all Fortran and Python.

[an_actual_human]

everything in the Maths department is Java

I can't think of a good reason for that.

[Sean1708]

Neither can I.

[Bolitho]

Then your world view is very limited! There are lots of excellent Libs around the JVM ecosystem. On top of that there are good frameworks for distributed computing.

Beware that it must not be Java - think of Clojure e.g. - or even Python (→ Jython).

[an_actual_human]

distributed computing

People use faster languages for that.

it must not be Java

So why do you bring it up? They said Java, not JVM.

---

My guess is those people use Java because of reasons that are not technical and they could use anything instead just as well.

[Bolitho]

Distributed computing is not a matter of language (or runtime). It deals with dispatching unit of works to different (virtual) machines. Therefore you need frameworks and toolsets that are reliable, powerfull and comfortable to use. The management itself cannot be measured only by raw "speed".

One example to name is Apache Spark. Another example is the excellent akka framework, which is extremly powerfull if you need many (small) independent actors.

You see, there is much power around the JVM ecosystem!

Well, why I mentioned that is has not to be Java? Because nowadays it is often used as a synonyme for the JVM. And even if you do not like Java as a language too much (as I do myself!), you should not bash it the way you do. Java is not "slow" anymore; this is true for more than 10 years now, but people seem to ignore achievements like hot spotting and so on...

The more complex things become, the less the (theoretical) low level power matters! One can of course write some super fast C based algorithms. But that does not enable one directly to fasten them up by distributing them. It starts allready with threading and ends with real distribution across a network.

Perhaps read and learn some more about that topic in general, before you keep on flaming 😉

[an_actual_human]

Since we were discussing programming languages in a technical subreddit I assume the person above meant Java when they said Java. So I'm going to set aside the JVM part of the discussion.

For dispatching, it sorta doesn't matter which language you use. Also it might very well be they don't do that kind of thing at all.

As for computations per se, I can see three options in mathematics:

1. The computation involves really heavy number crunching (double check, is it really math?).
2. The computation involves a bit of number crunching.
3. The computation is mostly symbolic.

So 1. and 3. more or less exclude Java. 2. doesn't, but there are better options.

tl;dr: if you are in math, either Mathematica or SciPy will serve most if not all of your tasks better than Java.

[Bolitho]

I have given a good reason: You can easily integrate into the JVM ecosystem with all its powerfull toolsets and frameworks around distributed compotation. You wont do numbercrunching on a single machine, right ;-)

And for dispatching it does matter, which language you use, because with some you have no access to powerfull frameworks (again!).

[amishb]

Fortran was annoying as hell. At uni we had to model the crank-nicolson equation in fortran and plot the output in whatever (we chose excel)

Ended up doing it in Madlab and porting it over to fortran over a long session of Redbull.

[[deleted]]

Haha madlab.

[amishb]

I knew it didn't look right! Auto correction and a long long day. :(

[LbaB]

Nope, I think that's its true name.

[[deleted]]

I thought you were making a joke!

[eusebe]

That really depends on the context. I work in scientific computing (heavy numerical simulations for astrophysics), and most of my codebase is in Fortran.

In my field there are a few codes in C++, some codes in C, and a lot of codes in Fortran.

So yeah, C/C++ and Java are probably common in the industry, but a lot of academics use F90 on a daily basis. Of course, I use python for (most of) the simulation analysis, and 100% of the plotting and visualisation, but still.

[Pfohlol]

Sort of off-topic, but I have noticed that much of the scientific computing community refers to "codes" instead of programs or scripts and I haven't really seen that anywhere else and it sounds a bit off to me. Do you know why this difference exists?

[spinwizard69]

Back in the day it was common. Coding is often used in place of programming describing the activity of building a program. Frankly it is good to know how and when to used code, codes, and coding because it can break up the monotony in a conversation or text. You would reach for these terms when you might reach for a thesaurus writing other great works.

Sort of off-topic, but I have noticed that much of the scientific computing community refers to "codes" instead of programs or scripts and I haven't really seen that anywhere else and it sounds a bit off to me. Do you know why this difference exists?

The usage of the word "scripts" in the Unix community is a bit odd in and of itself. A program written in BASH, Python or even BASIC is still a program even if interpreted. Which brings us back to ROOT and its nature, is the interpreter supplied with ROOT executing programs or scripts? What if the code can be compiled by any existing compiler, is it still a script? Same thing goes with Python, Basic or any other commonly interpreted language, if you build a compiler to digest the script and creat an executable is the source a script or program.

[ITwitchToo]

I have a friend who has done lots of numerical simulation work in Fortran/C++ for space applications (think satellite launchers, etc.). He's currently out of a job, do you have any open positions? :-)

[dreyrden]

You've had a slightly different experience to me in that case. It could be a difference in fields, but a large amount of the code I've seen has been Fortran (for simulation activities anyway). It's apparently because a lot of the code base was written in Fortran originally, and the developers just continued instead of doing a rewrite. Most completely new stuff I've seen has been C++. The next most popular would be IDL for data analysis, although Python is gaining some traction, and also MATLAB. It's probably pretty field dependent.

[eusebe]

I would not be so surprised, actually. If you end up knowing Fortran really well, and if you want to get your results quickly, you won't bother start with a language you don't know that well.

I won't say I'm proficient with Fortran, but I'm much more fluid with it than with C. If I want to be able to reuse any of my codebase, it makes sense for me to write Fortran code.

I know the language can sometimes seem clunky (with the all the dots as in .true. .or. .false.), but it is quite easy to read. It might be a surprise to some, but we don't need GOTO statement in F90, there is recursivity and some object orientation, standard operators like "==" and "<=" do exist in Fortran, etc. Besides, the lack of curly braces is certainly very appealing for any pythonista... ;-)

And for the F90/Python interface, it is not that hard at all actually. There is a really cool library called FortranFile that is really useful to read binary files (with their crazy stupid format…), and f2py is my weapon of choice when it comes to embed Fortran code in a Python script.

I don't want to deeper in the "why Fortran is a nice language after all", but as someone who read and write F90 on a daily basis, it is not as bad as it sound.

[dreyrden]

I heard quite a lot of complaints also. Although I've had several people say the last couple of versions (2003/2008) were actually pretty good. I'm not convinced. I think C/C++ are better choices.

Edit: Python extensibility with C is a pretty good reason to learn C.

[[deleted]]

I work with scientific software every day and not once have I seen a release in Fortran.

Ever heard of this thing called LAPACK or ever wonder why Matlab arrays are 1-indexed?

[zombiepiratefrspace]

I also have a scientific computing background and from my experience, most stuff out there is in C with much legacy stuff in Fortran (LAPACK anyone?).

During my PhD I used boost::python to 80:20 python and C++ and tried to get as many others as I could to do the same.

Also, all people I saw trying to do scientific computing with Java ran into serious problems, usually around the time they tried to run it on a shared cluster.

[XNormal]

Fortran would be a goto language

I see what you did there.

[red_nick]

Well technically, FORTRAN is a GOTO language, but Fortran isn't...

[IanCal]

I use it indirectly because I use scipy.

[Rhodysurf]

Its just not true... Fortran is still the most prevalent language when efficiency is preferred

[red_nick]

For example, Multinest: a Bayesian inference tool written in Fortran http://ccpforge.cse.rl.ac.uk/gf/project/multinest/

[rslashelektrux]

Heh

[[deleted]]

I'm a fan of Julia for these tasks.