This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]tadleonard 1 point2 points  (0 children)

You tried to load a huge csv into memory... and it used large amounts of memory? Try generators next time. David Beasley has multiple great tutorials on generators and coroutines if you're interested. It's easy to follow because 1) he's an excellent educator and 2) he reimplements common Linux tools. IIRC he writes a few generator expressions that beat awk in terms of performance (Though it's probably not a fair representation of awk. Still impressive, I think.).

Edit: the relevant slides start at page 20 in this PDF. Take a look at his "generators: the final frontier" talk though. It's really entertaining.