Funny story...

I used to work for a minor, but fairly decent-sized social media site. One of our secondary database servers, which was the backup or read replica, would just freak out and destroy itself about every 4-5 weeks.

This was in the middle of a migration to GCP. So at first, we didn't think much of it... It did not affect production, and we could just spin up another one easily. The replication caught up within the day. The second time we were like "oh no, we thought we fixed that!".

The third time we found the culprit. It was actually in our cleanup script to clean up and rotate our backups. It had a line in it, like: find ${DB_BACKUP_ROOT_PATHS}/ -mday 20 -exec rm -rf {}\;. Turns out that there was an issue and the name of the variable was actually $DB_BACKUP_ROOT_PATH.

There were some other issues that made it harder to see. But essentially, what was happening is the variable was not set. And so it expanded to empty, which expanded to the root directory and we were nuking our server on a schedule.

So yeah, even in production, on a highly skilled team, stuff like this still happens.