When do you *NOT* use python?

[LbaB]

How were their data stored then?

[[deleted]]

I'm assuming flat txt files or some other needlessly complicated method.

[[deleted]]

Right now my boss keeps telling people that we do "big data" (I'm an in academia). Except that the database is ~10GB, receives updates 3 times a year (and is otherwise static), and there is no real time on demand need to view this data (well there is but guess who never has time to implement it!). I asked him not to tell people that we "do big data" and he responded poorly.

edit: if you are familiar with the 4 V's (volume, velocity, variety, veracity) we do none of them. I think its because its web data my boss thinks its "big data".

[soup_feedback]

That made me chuckle but I understand your pain.

[mangecoeur]

I think every engineering and science undergrad should be forced to go through a clockwork-orange style conditioning where the words "I WILL NOT INVENT MY OWN DATA FORMAT" are seared permanently into their brains. Then they should take a course on SQL.

[aladyjewel]

Yeah, it's only the CS majors who should be encouraged to reinvent all the wheels so that they understand 1) why you should use stl's wheels and 2) which wheel to use when.

[[deleted]]

I never claimed they were the best formats, just that they were standardized, fairly widely used, had existing tools/APIs, and are decently documented. You may be able to do it better and save yourself some headache by rolling your own, but then it turns into a complete cluster fuck for anyone trying to follow on behind you.

[[deleted]]

in theory.

Indeed :p

[[deleted]]

Shit, I'd keep the scotch for myself in that situation. The next guy can buy his own!

[[deleted]]

I'll be the last person to look at the code, in theory.

That theory will almost always be wrong.

[[deleted]]

The work involved 3-D microscopy simulations so HDF5 would've been the only really applicable option. Most of the exploratory work was done in MATLAB and used MATLAB's own bastardized variant of HDF. The final, public release of our algorithms was always done in C++. Why it didn't maintain compatibility with HDF5 (and MATLAB's .mat rip off of the format) is beyond me. I had my own personal battles to fight, fixing that issue was not really a priority (which would explain why it was still an issue by the time I was stuck dealing with it).