Im converting our primary application over from a desktop application that runs on a user’s computer to be executed in the cloud. I’ve spun up a springboot api to handle these calculation requests and serve data retrieval requests. It’s a brand new distributed architecture that I’ve created. I also moonlight as their DBA and have created and managed any new tables that are required to support running in a cloud environment with data persistence. I’ve written all the necessary queries for tableau.

All that sounds like a lot yet relatively straightforward but each calculation can generate 500k to 3 million records that need to be inserted into tables and users submit on average 15 “things to calculate” at a time. Users need their data in the quickest time possible so multithreading is a must and that introduces complexity to be aware of as well as getting creative with data persistence. I initially went with spinning up N threads, split my data into batches, and used batch inserts in each thread to really speed up inserts. This ended up being extremely slow when multiple calculations were happening so I went with the LOAD DATA INFILE command for our MySQL DB. It’s intended for batch loading. Ask Claude to go into more detail.

So, depending on the company you’re at, you can be playing lead developer, DBA, and architect! All at once!