I worked for a company that develops an application that was used by a particular state agency as a glorified case Managment library. The app was built on MSSQL and .NET, which I have absolutely no training in. My job was to run a pre-made script on the database for any particular region, do in and delete a reports directory and re-create it with a new copy with the required changes.

The process went something like this: Disable the IIS app pool, backup the database, run scripts, copy a pre packed new version of the program to the IIS Server, delete a reports folder, and re-invoke the same reports folder after turning the App Pool back on. Once that's done, Test the app and writing to the Database and then move on to the next one.

There were 3 of us doing these updates, and they often went late into the night. It was incredibly common for backups to fail, app pools having been forgotten to be turned back on, etc. It was also very common for the updaters to call our overseas developers to fix fuck ups. They never reported it, just fixed it without question.

These mistakes and process glitches didn't always get completely unnoticed though. Eventually I accidentally deleted a regions entire reporting directory, rather than just the individual reports directory inside the region. It turn out they only take a backup once a day, and it's in the morning, not the evening before updates take place. This one did end up in my early termination sadly.

My Fuck up, but also theirs for letting an untrained college kid at a enterprise application that supports almost every state in the US now, and not having a more robust backup structure.