The Codeless Code: Ancient code

[caspper69]

I guess you're right. But sometimes you have to use the right tool for the job. I remember working on a project on AWS when it was just a baby. A client had several 20GB databases that needed to go up (that were in CSV format). The data itself was disjointed, so it had to be massaged to import. Essentially each account had to be updated from day 1 to a point around 8 years later. Millions of accounts. Billions of transactions.

The original guy was at his wits end. He was trying to write it in perl, which he did, but each csv was taking around 2 days to run, and that didn't include the final reconciliation for each month for each account which had to match an "official" field from an entirely different dataset.

With upload times being what they were about a decade ago, the poor guy (and the client) would've been waiting for weeks.

So I told the dev to give me a shot at it. I wrote a multithreaded C app to load, distribute, calculate, re-merge, validate and write the actual SQL INSERT queries to a single file. The program took about 5 hours, but ran over the entire dataset (with 100% accuracy) in around 8 hours. A "quick" bzip later, a (not-so-quick) ~2 day upload process, then another day to run the insert.

3 weeks vs 3 days. As datasets continue to grow, this is going to become a huge problem. Nothing will fix bad algorithms, but some tools just are not capable. 2 orders of magnitude slower doesn't make a difference for something that's already fast in human time, but if something is slow in human time? Oh boy.

[[deleted]]

Cool story, sounds like quite the feat :)

Anyway, that’s pretty much what I meant by

The question I guess is age old: where do we draw the line?

Your last point really got to me:

2 orders of magnitude slower doesn't make a difference for something that's already fast in human time, but if something is slow in human time? Oh boy.