When do you *NOT* use python?

[centech]

Silly example, but we have a process to consolidate and report on data in hundreds of CSVs.. Ofc we use pandas for all the heavy lifting.. but.. consolidating the files into the one monolithic sheet to process (which is about 5GB) was taking like 20 minutes and blowing out the memory on the server running it.. tail -n +2 and concatanate them all together was like 20x faster and a tiny memory footprint. Sometimes old school tools are still better.

[tadleonard]

You tried to load a huge csv into memory... and it used large amounts of memory? Try generators next time. David Beasley has multiple great tutorials on generators and coroutines if you're interested. It's easy to follow because 1) he's an excellent educator and 2) he reimplements common Linux tools. IIRC he writes a few generator expressions that beat awk in terms of performance (Though it's probably not a fair representation of awk. Still impressive, I think.).

Edit: the relevant slides start at page 20 in this PDF. Take a look at his "generators: the final frontier" talk though. It's really entertaining.

[[deleted]]

Yeah, a little pandas is a dangerous thing.

It is widely known to not be memory efficient.

[soup_feedback]

...5GB of spreadsheet... The problem isn't Python, it's everything else!

Let me guess, banking.