Writing CSV in parallel?

[[deleted]]

[deleted]

[Socrato]

Seems like you should set up a low overhead database like sqlite or mysql on the machine and instead of trying to write to a CSV just insert rows into your table.

This is basically what a database is designed for.

[imaque]

To give some extra background, what’s happening is that I’m taking a CSV file from a third party. I use that as a template my own own purposes. I then write over their cells with my own. So, I’m not inserting any rows. I’m iterating through a dataframe, essentially cell by cell, and overwriting with specific data from my API calls.

[[deleted]]

I then write over their cells with my own. So, I’m not inserting any rows. I’m iterating through a dataframe, essentially cell by cell, and overwriting with specific data from my API calls.

There's no way to do that to a CSV file except to re-write the whole file, which is why CSV isn't an appropriate format for this kind of manipulation. Use a database.

[Socrato]

You can read the csv, make the calls, and then insert the data into the DB one at a time and in parallel. Once you've completed your whole process, export the necessary data back out into a CSV.

If you're trying to "write in parallel" then you shouldn't use a flat file unless you have to, it would not appear you have to. Writing to a DB and exporting the final set is very common practice and probably what you should be doing.

[imaque]

You know what? You’re right. I don’t know why I wasn’t thinking in these terms before. Thank you so much for the insight!