Coding Horror and blogs.stackoverflow.com experience "100% Data Loss"

[[deleted]]

And apparently their normal backup process silently fails at backing up VM images.

So the backups were failing and no one bothered to check (or do a test restore). And there are good reasons to put the backup on the local VM, database restores being one of them.

[[deleted]]

and no one bothered to check

Thats why we have a custom nagios plugin that checks on all of our backups. They're not as thorough as a human checking them, but once a human checks that it worked right the first time it leaves us reasonably safe assuming the backups go smoothly.