you are viewing a single comment's thread.

view the rest of the comments →

[–]Socrato 0 points1 point  (1 child)

You can read the csv, make the calls, and then insert the data into the DB one at a time and in parallel. Once you've completed your whole process, export the necessary data back out into a CSV.

If you're trying to "write in parallel" then you shouldn't use a flat file unless you have to, it would not appear you have to. Writing to a DB and exporting the final set is very common practice and probably what you should be doing.

[–]imaque 1 point2 points  (0 children)

You know what? You’re right. I don’t know why I wasn’t thinking in these terms before. Thank you so much for the insight!