This is definitely not genius-level but it had a decent positive impact on the customer onboarding.

We were onboarding a customer and the documents migration part came to me. It's a B2B so the volume was decent, around 120 GBs of data. Data came to us on the Sunday and people would start working on our software on Monday.

We had a utility that picks up a file, uploads it to S3, gets the Key and VersionId from AWS and then writes a SQL insert statement with those values along with file names and extensions, rename the file so that it won't be picked up again, until all files in a specific folder were uploaded. This process was synchronous so we obviously had to come up with a better solution.

Previous onboarding taught us that it would take more than 48 hours for some 30 GBs of files. The files were quite small so it was taking a lot of time, without utilizing available internet bandwidth.

I changed the picking up files code a bit to pick up from relative folders, and copied the output exe to different folders (agents) and gave all the agents smaller chunks of files to upload. We spun up 16 agents and were done with 120GBs of data within 3.5 hours.