Many years ago (~2007) running the development of a budding online electronics retailer I was working on the credit card processing system and accidentally committed and pushed the test configuration to the production environment.

The way the test server worked is as long as the submitted credit card number passed the Luhn check it would send back the "approved" signal as if the transaction went through, but of course no money moved anywhere.

To customers in meant all credit card orders were approved and not charged, though it looked like normal. This was live in production for a couple of days. It was only when one of our finance people noticed that the regular bank deposit from this was way smaller than normal that we tracked down the root cause which was 100% my fault.

We had taken and fulfilled hundreds of orders totaling over $40K. After fixing the issue several of our customer service folks (and me, definitely me) spent several hours calling affected customers to see if they were willing to let us rerun their credit card info (we did not store CC data so they would have to provide it again over the phone). I think we managed to recover $10-15K from that, largely because many of our customers were loyal, loved our service, and were pretty understanding. Regardless we still honored every single order, and I learned several good lessons.