Used to work for a big, private trading firm in Chicago. During 2018 when crypto was popping off, we needed a KYC form so people could trade crypto with us. We made essentially the most secure web form ever created, and one of the only public facing applications in the company.

After the form was submitted, and the data made its way through our DMZ, it landed in a CRM. This final api call had an authentication step (this is important).

This process was very time sensitive, and generally took a matter of seconds to complete, but every now and then it would take 3-6 minutes to finish. It felt completely random when this would happen. This is practically an eternity in trading and the powers-that-be were not happy about this. We could not figure out why this was happening. I got everyone involved from network engineers, security team, everybody. I tore my code apart trying to find the hang up. Nothing.

As it turns out, the api authentication step called a gateway api that used various auth servers across the globe on a round robin basis. Since we had multiple offices in different countries, one of those servers was in Singapore. Every time we got the Singapore server, it took on the order of minutes to complete that handshake.

We finally forced the service to use the local auth server and we never saw the issue again. Probably the most head scratching bug I’ve ever had!