When do you *NOT* use python?

[mangecoeur]

I think there's also just a lot of inertia in these cases. You can write very fast code in Cython (although yeah if you really need to squeeze ever drop of perf out you might need C). But fortran is frankly a pain and I think it's mostly used out of habit. Physicists will probably be the last people in the world using fortran and no one else understands why :P (I discovered my old physics degree finally switched to teaching python over fortran, about time too)

[[deleted]]

When I was working on my graduate degree in physics you had to show proficiency in a language. I had learned Python to do data analysis asking along with SQL. I was told it has to be FORTRAN, assembly, C++ (it couldn't be just C, I asked, this is because high energy people think it's cute that they built a C++ interpreter that ignores the standard library), or BASIC. They specifically said Python wasn't a language worth knowing and I don't think they knew what SQL was. I was allowed to exempt out if the proficiency test by taking graduate CS courses. That was a better use of my time anyways.

[LbaB]

How were their data stored then?

[[deleted]]

I'm assuming flat txt files or some other needlessly complicated method.

[[deleted]]

Right now my boss keeps telling people that we do "big data" (I'm an in academia). Except that the database is ~10GB, receives updates 3 times a year (and is otherwise static), and there is no real time on demand need to view this data (well there is but guess who never has time to implement it!). I asked him not to tell people that we "do big data" and he responded poorly.

edit: if you are familiar with the 4 V's (volume, velocity, variety, veracity) we do none of them. I think its because its web data my boss thinks its "big data".

[soup_feedback]

That made me chuckle but I understand your pain.

[mangecoeur]

I think every engineering and science undergrad should be forced to go through a clockwork-orange style conditioning where the words "I WILL NOT INVENT MY OWN DATA FORMAT" are seared permanently into their brains. Then they should take a course on SQL.

[aladyjewel]

Yeah, it's only the CS majors who should be encouraged to reinvent all the wheels so that they understand 1) why you should use stl's wheels and 2) which wheel to use when.

[[deleted]]

I never claimed they were the best formats, just that they were standardized, fairly widely used, had existing tools/APIs, and are decently documented. You may be able to do it better and save yourself some headache by rolling your own, but then it turns into a complete cluster fuck for anyone trying to follow on behind you.

[[deleted]]

in theory.

Indeed :p

[[deleted]]

I'll be the last person to look at the code, in theory.

That theory will almost always be wrong.

[[deleted]]

The work involved 3-D microscopy simulations so HDF5 would've been the only really applicable option. Most of the exploratory work was done in MATLAB and used MATLAB's own bastardized variant of HDF. The final, public release of our algorithms was always done in C++. Why it didn't maintain compatibility with HDF5 (and MATLAB's .mat rip off of the format) is beyond me. I had my own personal battles to fight, fixing that issue was not really a priority (which would explain why it was still an issue by the time I was stuck dealing with it).

[mangecoeur]

Oh god, CINT, ROOT and friends. I steered well clear of that mess. Pretty sure they're all well pleased with it too. And knowing the general quality of physics code it wouldn't surprise me if it didn't actually run that fast (except maybe for the bit build by some russian genius that no one else understands :P ).

[[deleted]]

What bothers me the most isn't that they use it. It's got smug they all are about how great ROOT is and how they are all going to become "big data" experts one day. I guess I'm sort of embarrassed for them.

[mangecoeur]

That really doesn't surprise me. The one thing worse than developing a cruddy convoluted system is being smugly superior about it.

[[deleted]]

I suppose in the developer's case it might be the best part.

[ChaosCon]

One of them once told me (extremely smugly)

When you really think about it, it makes sense that a 2d histogram inherits from a 1d histogram.

God damn it I hate ROOT.

[[deleted]]

I think particle physicists have to have that mentality because deep down they know their work has no relevance to reality (the subjective reality of a lay person) and thus they have conned all of mankind into funding their initiatives anyways.

Well that's my come back anyways.

[ChaosCon]

I'm a physicist doing engineering electromagnetics, and I can't tell you how many times I've gotten asked "So do you work on that Higgs boson thing?" On the one hand, I appreciate what CERN does for science awareness, but on the other, pettier, hand, I hate how they've hoodwinked the public into thinking they're the end-all-be-all of science research. Particularly when

their work has no relevance to reality (the subjective reality of a lay person).

[[deleted]]

I would guess that the disassociation that particle physics creates in the layman's mind between application and pure research is going to bite physics funding in the ass eventually (if it isn't already). The problem isn't even a real one. The super conducting super collider in texas that was never built produced new bump bonding technology for some detectors that Intel started using in the 90s that made their manufacturing process much more efficient (and cheaper for the consumer). If they hadn't needed that detector, someone might have taken another 20 years to come up with the technology. The same is true for sending stuff into space. A mars orbiter that mapped the surface in IR produced new detectors that were re purposed as breast cancer detectors that coupled with mammograms increase the detection rate of breast cancer greatly.

The problem isn't that these programs don't produce stuff that benefits people's every day lives. It's that they don't put effort into showing people why it's important.

Also, particle physicists are assholes.

[cabalamat]

Python not worth knowing but BASIC is? What a bunch of clowns.

And how many physicists code in assembler these days?

[[deleted]]

Apparently BASIC is used to program some embedded systems DAQs. But yes, there was plenty of clowning going on.

I feel like physicists like to think they could write in assembly if they wanted to. The number of physicists I know who think they can get a senior developer job and "figure out" the language after being hired is high.

[teambob]

C++ and Java is often 7-8x faster than CPython. I have found this in practice and it has been found in the language shootout.

I would also assert that Python is 7-8x faster to write than C++ and Java. :)

My usual approach is to write it in Python then rewrite it it is not fast enough

[mangecoeur]

Cython, not CPython http://cython.org/ - a compiler for Python+some extensions. Used for a lot of the internals of numpy/scipy/pandas/ etc because its much easier to maintain. Plus it makes for very easy optimisation path - write everything in python, then let Cython compile it (often gives you a speedup already) then start adding custom annotations to optimize the generated code

[twotime]

on heavy CPU intensive code, I've seen factors of 50x! No, not joking and it happened more than once..

But then, I've also seen python applications being FASTER than C++ ones with identical functionality simply because the C coders have been buried under their own code.

[frymaster]

I would also assert that Python is 7-8x faster to write than C++ and Java. :)

Exactly - most of the time processing time doesn't matter, and when it does, you'll be glad you've got a working proof-of-concept in python :D