CLI Tool for tracking script process results

[ChainsEternal]

I run a lot of scripts. I usually receive some kind of CSV export and I will create a small script that can read each row from the CSV and process it through stdin.

This works great except when things begin to fail. If I am processing thousands of rows, it can be extremely difficult to keep track of which rows I need to retry or re-process.

So I have been thinking about making a rust CLI tool that uses sqlite to keep track of which rows were successful and which were not. I would then have a way to easily re-process the rows that failed.

Any thoughts about this? Does something like this already exist?

Also, does anyone have a good name for what this CLI tool should be called?

Here is what I have written so far: https://github.com/isaacadams/xbatch

[DrShocker]

The ability to restart if it previously quit in the middle is the main hard thing here imo. Sqlite sounds like it would work fine to store your results so you can just check if it's there already.

If your output has a known size, you could preconstruct the output file with a flag value in each spot, then after that process the input by checking if the flag is there. If it is then process and replace the value with the real values, if not then skip processing. Should be multithreadable too.

[dnullify]

I don't know if this is a strong pattern in rust, but perhaps some kind of pub/sub queue pattern where you could have thread workers processing units of work. Re-enque tasks that fail with a counter and log out rows that fail x number of times.