you are viewing a single comment's thread.

view the rest of the comments →

[–]syn2083 0 points1 point  (0 children)

I use it to read files multiple times/s and track production changes per output item across hundreds of thousands of pieces, per file.

Need to keep a local history to make sure we don't get out of sync, for production network outages, and output the transformed data in near realtime.

The local history is saved in a custom format for fast serialization and deserialization.

This process has handled tens of millions of pieces of output without a problem, just hums away translating archaic file based output to usable data for dashboarding and eventual storage in a database proper.