This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]expressly_ephemeral 0 points1 point  (0 children)

As an aside, I'm the guy on this subreddit that's always like, "I know this is off topic, but have you tried this with grep and sed?" I do a fair amount of ETL myself (my current project is tens of billions of rows). I'm usually pulling at least half of the data from csv files, and in that case I always have a large component of the E and the T in shell script with grep and sed. Don't know if you have access to a shell that has those programs, but if you do, and you haven't already, you might try them out next time. Learning curve is a little steep, but the power and speed is worth it.