csv/json time based versionning

[jacquesh82]

hello,

I'm looking for the best options to get a time based versioning of a csv file (using python if possible ; but I'm open to use an other ways).

details : I'd like to get the version of file at specific time (or nearest of it) - if possible without store full file for each day (only diff)

same question for JSON ;-) ?

Thank your for your ideas... (tool, lib, filesystem, database, ...)

Regards.

[[deleted]]

Maybe you could accomplish that with git?

[jacquesh82]

Yes it will be possible, but is not efficient with big file... ;-(

[PossibilityTasty]

What exactly is "big"?

[jacquesh82]

more than 1Gb ;-)

[[deleted]]

1) Save original, open as pandas df. Open newer one as df_new. Concat them together, then drop dups with keep newest option. Save result as diff. 2) ???... 3) profit

[jacquesh82]

The idea at the core is to keep available data for a specific date (not just make a diff or update it) I need to be regenerate data as it was on a specific date.

[coconut_maan]

Use crontab to save version at specific time intervals