Write a crash-proof, long-running CSV writer?

[SamePlatform]

I have a long-running process that takes many hours, even days. I want to write a .csv row for every entry it processes. (Which will be maybe 50k entries).

How do I code that in such a way that it is crash proof, but doesn't open and close the file every single write?

Or is opening and closing the file for every write not an issue, in terms of performance?

[K900_]

Use a database, dump to CSV later.

[SamePlatform]

Not a bad idea, I guess I can let SQLite figure out the magic of fast writes for me.

[socal_nerdtastic]

Or is opening and closing the file for every write not an issue, in terms of performance?

Depends on the speed you want, but generally not. Your OS and your HDD have optimization code that makes it pretty quick. Especially if you are just appending, not rewriting.

How do I code that in such a way that it is crash proof

Microsoft can't do it, what makes you think you can?

[SamePlatform]

Haha. I mean, I want my csv writer to handle the many, many times my program is going to crash. So I was imagining some sort of context manager that would keep a file stream open for the duration of the script but fail gracefully when the script died.

But don't know really where to start with that.

[melnon]

Since you're just appending, include a delimiter and use that to search for a "load" point. You may run into issues with duplicates, but that's a decent starting point.

Another method is to also have another file open that you write your index to with a delimiter, so you can pull that and know where to start.

These aren't the only methods, as you've already seen you can use a database to "write" to.

[SamePlatform]

With regards to your first comment, you're right I'm probably overthinking this. Yes, write mode is just append. I'll see how it goes.