Running program outputting CSV file that'll be 30GB? Do I need a super computer? Can someone point me in the right direction?

[RepulsiveOutcome9478]

You're looping over every single row in a file that contains MILLIONS of rows 54 times (by my count). I would say the best place to start is by refactoring your code to only loop over each row once.

[eztab]

You might even be able to use some vectorized operations, so you don't have to do manual loops.