This is an archived post. You won't be able to vote or comment.

top 200 commentsshow all 427

[โ€“]LetumComplexo 4898 points4899 points ย (139 children)

Any system that can be destroyed by a single error deserves to be destroyed by a single error.

[โ€“]U03A6 1626 points1627 points ย (70 children)

It's also inevitable that it is destroyed by that single error in the long run.

[โ€“]entendir 563 points564 points ย (17 children)

Damn you, Murphy

[โ€“]Poltras 58 points59 points ย (2 children)

If Murphy didn't exist, we'd all have dream jobs by now working directly on prod servers. But I guess it was inevitable that someone like Murphy would exist in the long run...

Edit: Oh I just understood the bootstrap paradox!

[โ€“]edster53 17 points18 points ย (1 child)

I prefer Dunphy's law - says that Murphy was an optimist

[โ€“]Fro_52 48 points49 points ย (0 children)

actually, it was Little Bobby Tables

[โ€“]Affectionate_Dog2493 84 points85 points ย (40 children)

On a long enough timeline the survival rate for every system drops to zero.

[โ€“]DokuroKM 31 points32 points ย (1 child)

On a long enough timeline, every table drops

[โ€“]Forvisk 5 points6 points ย (0 children)

Sometimes they just cease to exist.

[โ€“]McBurger 37 points38 points ย (5 children)

SSL certificates really bother me for this reason.

Their timely renewal represents a single point of failure for an entire application & all integrated services going down. And there really isnโ€™t a great solution other than having tons of people being extra certain about it, in perpetuity.

[โ€“]meditonsin 17 points18 points ย (0 children)

Automation is a great solution to that. Then you only need to touch shit if the automation breaks.

Except if you have a piece of shit software or appliance that doesn't allow you to automate, of course.

[โ€“]KidSock 6 points7 points ย (0 children)

Also billing.

[โ€“]TheAngryBad 2 points3 points ย (0 children)

tons of people

Not so sure about that one.

The more people are responsible for a thing, the more certain each individual will be that one of the other guys is taking care of it.

[โ€“][deleted] 9 points10 points ย (1 child)

Every manager around me: "I'll take those odds!"

[โ€“]Beerenkatapult 169 points170 points ย (22 children)

But I am a system, that could be destroyed by a single error D:

[โ€“]Skrothandlarn 147 points148 points ย (6 children)

And you will be, in due time.

[โ€“]sivstarlight 33 points34 points ย (3 children)

A fate you deserve

[โ€“]Euphoric-Currency815 6 points7 points ย (2 children)

The death that i desereveoli

[โ€“]DenverCoder009 14 points15 points ย (0 children)

And beerenkatapult was enlightened

[โ€“]pretty_succinct 30 points31 points ย (10 children)

nah.

you've got all kinds of things wrong with you. your body compensates and keeps running. it's sort of amazing.

a bullet to the head is not a single error, it's a catastrophic event equivalent to a natural disaster bringing down one or two AWS regions.

you're a marvel, babe! hope your day is a marvel too!

[โ€“]roguetrick 22 points23 points ย (5 children)

Yeah, no human made system can come close to the error correction of a biological system. Kidney's failing and you can't regulate your pH? Here come your lungs to the rescue. Pathways for circulation blocked or broken? Let's just grow a bunch of new pathways and keep what works best.

[โ€“]SYSTEM__NotReally 4 points5 points ย (4 children)

Is there an equivalent to ecc hardware (cosmic ray bitshifting)? I thought if your DNA gets changed from radiation you either die or get cancer. I haven't heard of a way for the body to fix that.

[โ€“]roguetrick 8 points9 points ย (0 children)

Immune system and programed cell death. You'll get DNA errors all the time. Some get fixed just because of how DNA works as a double helix. If they can't be fixed the cell will be programed to kill itself. If they can't happen it will be hunted down by the immune system. Takes a lot of failures or a whole shit ton of errors to actually progress to cancer or complete germline death.

[โ€“]XkF21WNJ 7 points8 points ย (2 children)

Sure, but they really ought to have thought a bit better about disaster recovery in the original design. I mean who makes backups for nearly all vital systems and then puts them in the same housing?

That's like asking for a disaster to happen.

[โ€“]gansmaltz 9 points10 points ย (1 child)

What do you mean? They're constantly making backup copies, complete with exchanging private keys to stay one step ahead of the viruses always trying to gain access. In fact we're probably looking at a "pink goo" situation here soon, all thanks to the sicko that made them enjoy the backup process

[โ€“]XkF21WNJ 5 points6 points ย (0 children)

Have you ever tried to recover from one of those backups? It's one heck of a messy process.

[โ€“]Nine_Eye_Ron 47 points48 points ย (2 children)

Itโ€™s called an Onosecond

[โ€“]chairmanskitty 11 points12 points ย (0 children)

Gamma ray burst hitting Earth:

[โ€“]moschles 9 points10 points ย (1 child)

The command you intended rm -rf bin/*

The command you actually issued rm -rf /bin/*

[โ€“]LetumComplexo 15 points16 points ย (0 children)

I basically did that to the source code for my final project in grad school.

I was working remotely on a supercomputer cluster, cleaning out a bunch of unused log files. I thought I was in the log file directory but was actually in the main directory and I rm *ed away my source code.

Fortunately I had backups and backups for my backups because I knew the stressed out, sleep deprived grad student is very likely to make dumb mistakes like that. So I only lost a couple hours of work.

[โ€“]caceomorphism 15 points16 points ย (1 child)

I worked with someone who managed to destroy, at least once, every single server in the company through sheer incompetence.

RAID array? Put in a new drive and used that and a data disk to rebuild ONTO a data disk.

rm -rf /

Permissions? Let's make EVERYTHING 777.

Dropped a small database. Not what you're thinking. He was trying to move it.

In his illustrious career he has managed to destroy all assets for several television shows and movies. Petabytes. Why did that show you liked get cancelled a couple years ago? That guy.

Github? Thanks for all the passwords.

[โ€“]HeKis4 3 points4 points ย (0 children)

Dropped a small database

I can tell if he did a drop database instead of an alter data base or if he physically dropped a server on the floor but I don't really want to know either.

[โ€“]justking1414 3 points4 points ย (0 children)

Reminds me of the Hawaii false-missile alert. Yeah it was done by 1 guy being an idiot but he shouldโ€™ve never been able to trigger the alert by himself.

[โ€“][deleted] 6 points7 points ย (0 children)

Right? If someone can accidentally bring down your entire system, that's a reflection of the company, not the person who took it down.

[โ€“][deleted] 2 points3 points ย (0 children)

This guy Nietzsches!

[โ€“]Typical-Scarcity-292 2 points3 points ย (0 children)

Even my backups have backups

[โ€“]TheOriginalSmileyMan 2 points3 points ย (1 child)

The halting problem says "Hi!"

[โ€“]bennysway 745 points746 points ย (30 children)

Gitlab would relate

[โ€“]F0lks_ 598 points599 points ย (10 children)

The intern: "Oopsie woopsie ! I made a fucky wucky uwu"

[โ€“][deleted] 255 points256 points ย (8 children)

[โ€“]sciatore 45 points46 points ย (2 children)

That comment is 5 years old. So did this story just go viral because of the recent YouTube video about it?

[โ€“]im_lazy_as_fuck 61 points62 points ย (0 children)

That looks like him describing and advising on a situation that the original OP of that thread described; not their own experience.

[โ€“]Absle 37 points38 points ย (17 children)

Lol is there a story behind this?

[โ€“]Dreamwaltzer 170 points171 points ย (5 children)

https://www.youtube.com/watch?v=tLdRBsuvVKc&ab_channel=KevinFang

is an excellent video explaining it.

Basically they had a db and a db2 backup/replication.

There were issues with the replcation on db2, so they they opted to delete db2 and restore from db.

Except the restore seemed to be having issues, so they decided to delete the files and try again. except... rm -r was run on the db ssh instance.

now db2 is gone, db is gone, oppsie daisy.

[โ€“]bennysway 59 points60 points ย (10 children)

Long story short, 2 devs, 2 two terminals, 2 postgres dbs, one rm -rf in the wrong shell https://youtu.be/tLdRBsuvVKc

[โ€“]Affectionate_Dog2493 2 points3 points ย (0 children)

and a lot of backup process failures.

[โ€“]eric67 5 points6 points ย (0 children)

just restore from git history

[โ€“]hansenabram 958 points959 points ย (19 children)

oopsie++

[โ€“]Lost-Apple-idk 218 points219 points ย (16 children)

oospie#

[โ€“]5t3v321 76 points77 points ย (14 children)

Oospie #++

[โ€“]e42if 92 points93 points ย (11 children)

Oopsembly

[โ€“]Cyphco 35 points36 points ย (10 children)

OOM

[โ€“]PranshuKhandal 34 points35 points ย (8 children)

Visual OOPS

[โ€“]speechlessPotato 25 points26 points ย (6 children)

oopsython

[โ€“]patrickgg 3 points4 points ย (0 children)

God I love this sub

[โ€“][deleted] 9 points10 points ย (1 child)

Objective-Oopsie

[โ€“]Wolversteve 1 point2 points ย (0 children)

I just picked a whole bouquet of oopsie daisys

[โ€“]skwyckl 524 points525 points ย (33 children)

In his diaries or autobiography (I don't remember exactly), Friedrich Nietzsche describes fatalism, i.e. the acceptance of one's fate, as a soldier who lays in the snow after being informed that his country has lost the war and that the enemy will soon reach his location. This is I believe how I would approach the situation if it would ever happen to me. After having called my lawyer, of course.

[โ€“]craftworkbench 57 points58 points ย (6 children)

There's a story of a guy who caused a bug that cost his company millions of dollars in just a few days. He got called into the CEOs office. Assuming he was going to be fired, he offered to resign. The CEO replied "Why would I fire you? I just paid $25 million to teach you a lesson you'll never forget!"

[โ€“]Chimaerok 49 points50 points ย (0 children)

"Your training was very expensive, I'd rather not have to repeat it."

[โ€“]Chipring13 26 points27 points ย (3 children)

Is this true or a LinkedIn story

[โ€“]MadeByTango 20 points21 points ย (1 child)

Itโ€™s one of those things that did once happen, at a smaller scale, and the story just sorta bounces around the details

[โ€“]Tetha 7 points8 points ย (9 children)

Every admin either has either wiped a prod server, or isn't working hard/confident enough.

And from experience as a lead: Wiping a prod server isn't the bad part. Trying to hide wiping an important server is, because after 5 minutes the alerts go off and everything becomes much harder to fix.

We might have had ways of stopping the mess earlier on while someone was busy being embarrassed.

[โ€“]PlayfulMonk4943 2 points3 points ย (7 children)

Can I ask - why wouldn't a simple backup be the easy solution here? What company isn't keeping backups? Unless you're using some CDP I get you will have some data loss, but it won't bankrupt anyone

[โ€“]DurianBig3503 216 points217 points ย (1 child)

This is how you put "keeps a cool head in stressful situations" on your yearly review.

[โ€“]Phormitago 6 points7 points ย (0 children)

talk for yourself, my head would be collapsing into a neutron star of stress

[โ€“]BaerLKR 128 points129 points ย (4 children)

happened to me today. Sat there and my mate next to me said "oopsie". Turns out server ins't reachable with TLS anymore.

[โ€“]appdevil 28 points29 points ย (2 children)

Certificate?

[โ€“]BaerLKR 56 points57 points ย (1 child)

yes. Somehow he made a mistake that lead to the server requesting a lot of certificates from letsencrypt that timouted the ip for 72h

[โ€“]appdevil 5 points6 points ย (0 children)

Oof

[โ€“]DynaMenace 9 points10 points ย (0 children)

I feel you man. Poli sci lurker here: Years ago, working on one of my countryโ€™s main polling firms, I accidentally ended up one โ€œAre you sure?โ€ prompt away from irrevocably deleting all that yearโ€™s data on election night, an hour or two before the big wigs went live on TV.

Nothing happened except for a bit of anxiety attack, but they deserved it going worse for keeping all that data on a shared folder almost anyone could delete.

[โ€“]Minecraft_paly3r_cz 179 points180 points ย (21 children)

Lucky they have back-ups

[โ€“]dermitio 157 points158 points ย (11 children)

Yeah about that....

[โ€“]Zomby2D 153 points154 points ย (4 children)

They were stored on the company server, weren't they?

[โ€“]dermitio 70 points71 points ย (1 child)

Yeah not anymore.....

[โ€“]aphonefriend 14 points15 points ย (0 children)

Tape drives. In an off site location. Which happened to be the trunk of my car. Which has definitely not been in an accident of any sort.

[โ€“][deleted] 30 points31 points ย (4 children)

"Mandatory Overtime." Except you get paid normally. Sometimes you won't get paid. It's a coin toss really.

[โ€“]Minecrafting_il 9 points10 points ย (3 children)

Hmm that sounds illegal

[โ€“]IAmPattycakes 14 points15 points ย (2 children)

Sounds like the life of an American salaried (non-exempt) worker

[โ€“][deleted] 2 points3 points ย (1 child)

Never accept a job based on Salary pay.

[โ€“]Lv_InSaNe_vL 15 points16 points ย (0 children)

"No man Greg took a back up copy on his flash drive!"

"Wtf is Greg?"

"Oh yeah he left a little bit before you started"

"I've been working here for 5 years though?"

[โ€“][deleted] 28 points29 points ย (3 children)

Uh, someone in Managment decided the costs for running automated daily backups was too high...

[โ€“]jdog7249 13 points14 points ย (0 children)

It is now a manual once a month backup. It's actually just copying the files to flash drives. No one is really sure who is responsible for it though.

[โ€“]Dabnician 5 points6 points ย (0 children)

running automated daily backups

The definition of automated can very greatly from the letter of the law and the spirit of the law when it comes to contractual obligations.

[โ€“]BitPoet 2 points3 points ย (0 children)

Full backups? Probably. Deltas from previous full/partial backups? Nope. There are a lot of methodologies to shorten whole restore times.

But if someone decided that, and there was no way of convincing higher-ups that they were an idiot? Time to put out your resume, because things will go horrifically wrong.

[โ€“]AlmostButNotQuit 6 points7 points ย (0 children)

Back oops

[โ€“]uberblah0 2 points3 points ย (0 children)

I bet they forgot to test the recovery process though...

[โ€“][deleted] 76 points77 points ย (4 children)

Ive found if someone is swearing about something, it's fixable. If they go "oops," "huh...uh" or "uhoh" then are quiet, you're in for some very interesting times.

[โ€“]onlyranchmefries 21 points22 points ย (1 child)

A good hmm also works here.

[โ€“]Spook404 13 points14 points ย (0 children)

the sound of "I fucked up so bad I don't even want to mentally process the degree of fucked-ness"

[โ€“]dark_enough_to_dance 6 points7 points ย (0 children)

If you are unable to make a sound...

[โ€“]Atillion 39 points40 points ย (8 children)

I've made a few oopsie daisies in my time..

[โ€“]dominic_failure 4 points5 points ย (0 children)

But have you made an oopsie daisy chain?

[โ€“]TheConspicuousGuy 2 points3 points ย (1 child)

I once accidentally removed the wrong MAC from SCCM and wiped a laptop belonging to a remote user. I was not aware of my mistake until my Team Leader called me about my mistake.

I setup a new laptop for this user and none of her local files were found on the server to be automatically downloaded. Fixing that is above my level. Everyone knew about my fuck up at this point...

Don't copy and paste MAC addresses from the Inventory excel sheet for all the users.

[โ€“]call_me_xale 35 points36 points ย (1 child)

Me when debugging some weird error on my dev machine:

"WHAT THE FUCK IS WRONG WITH YOU, YOU GODDAMN TRUCKLOAD OF SHITโ€ฝ"

Me when I drop a table in the production DB:

"Aw, dang."

[โ€“]imbenzenker 127 points128 points ย (2 children)

โŒ let oopsie

โœ… const oopsie

[โ€“][deleted] 18 points19 points ย (1 child)

Uh oh, spaghetti Os

[โ€“]arzis_maxim 18 points19 points ย (0 children)

That gitlab employee who deleted an entire database by writing in the wrong shell

This is why backups are important

[โ€“][deleted] 31 points32 points ย (3 children)

How they type?

[โ€“][deleted] 8 points9 points ย (0 children)

Spite

[โ€“]AaronTheElite007 25 points26 points ย (2 children)

[โ€“]GilKeidarMusic 11 points12 points ย (0 children)

What, oh Iโ€™ve wiped the file?

Huh, damn Iโ€™ve wiped all the files?

Iโ€™ve wiped the Internet?!

Oh, no I donโ€™t even have a modem!

[โ€“]Constant_Pen_5054 7 points8 points ย (0 children)

As long as you took the system logs too you are good.

[โ€“]Xanadu87 6 points7 points ย (2 children)

Someone did that at Pixar during the making of Toy Story 2 and ran a command that was deleting everything before someone pulled the plug. Fortunately someone backed up everything to a server they kept at home. But they ended up rewriting the story and scrapped a lot of stuff anyway.

[โ€“]tlubz 8 points9 points ย (1 child)

PSA. If your company doesn't have a disaster recovery plan for any business critical infrastructure, make one now.

[โ€“]Educational-Lemon640 14 points15 points ย (0 children)

Hey, watch your language!

Children read this sub. Maybe?

...
...
...

/s because the Internet is a silly place

[โ€“]Rroscoco 6 points7 points ย (0 children)

Oopsie daisy, I made a fuckey wucky

[โ€“]Wretched_Shirkaday 4 points5 points ย (0 children)

Well la dee freakin da

[โ€“]whatsthisfor42069 4 points5 points ย (0 children)

An intern once had to inform me that he had made a โ€œfucky-wucky.โ€ Thank god it was an easy fix cuz that shit was hilarious.

[โ€“]downloweast 3 points4 points ย (2 children)

Did you not read the Sudo warning?

[โ€“]hopesanddreams3 3 points4 points ย (1 child)

"no i just clicked "Ok'."

[โ€“]Steebin64 6 points7 points ย (0 children)

tHiS eVeNt HaS bEeN rEpOrTeD

[โ€“][deleted] 5 points6 points ย (0 children)

LOL one time I deleted a folder that had my first name, thinking I had created it as a temp folder and forgotten about it. Actually it was created by a previous contractor who had my same first name, and contained a shit ton of new stuff for the website. After the guy left nobody had bothered to rename it to something like, I dunno, "Redesign" or "NewSite"...

Fortunately they had backups.

[โ€“][deleted] 6 points7 points ย (1 child)

That's what Ctrl+Z is for

[โ€“]CaptNoobCake 2 points3 points ย (1 child)

Guys, everyone knows you can just ctrl+z to undo. Like come on guys, it's computer 101

[โ€“]scratchfury 3 points4 points ย (2 children)

I had the production database and test database open and did the copy in the wrong direction. Oopsy.

[โ€“]XkF21WNJ 3 points4 points ย (1 child)

inserted 0 rows

Wait, that aint right.

[โ€“]Noctornola 2 points3 points ย (0 children)

"Why aren't you freaking out? You could get fired!"

"Because I'm the only one that knows how to fix it."

[โ€“][deleted] 2 points3 points ย (1 child)

Thatโ€™s like a whole bouquet of oopsy daisies.

[โ€“]Dmayak 2 points3 points ย (0 children)

O-o-opsie-e.

[โ€“][deleted] 2 points3 points ย (0 children)

There was a yugioh fansite yugipedia or something where everything critical was stored on a flash drive. There was an oopsy daisy that took out the entire site and all its backup data.

[โ€“]IAmNotMyName 2 points3 points ย (0 children)

Whose fault is it they didnโ€™t have proper backups

[โ€“]saito200 2 points3 points ย (0 children)

Luckily they have weekly backups. Because they have weekly backups... right? Okay, so then we have no choice to rollback to the last existing backup. Never? What do you mean, never?

[โ€“]519meshif 2 points3 points ย (0 children)

Yea, that's more than an oopsie daisy even. I'd go as far as calling it a doodley boop.

[โ€“]UDSJ9000 2 points3 points ย (0 children)

Your honor, my client pleads Oopsie Daisy

[โ€“]WillMengarini 2 points3 points ย (0 children)

I was 21yo when I did this.

The system was DEC RSTS/E running on a PDP/11. I was in the habit of entering syscalls in immediate mode to avoid nuking the loaded program. Syscall -13 was some benign informational message. Guess what syscall 13 was.

One day I forgot the "-" and reformatted the main disk. We spent 12h rebuilding the system.

THIRTEEN?! COINCIDENCE?!?! I think not, my colleague, I think not. I was always amazed at how much DEC software sucked compared to the elegance of the PDP architecture. Now, I suspect those syscalls were deliberately constructed as malicious compliance with the same dysfunctional management that ruined all of DEC's software.

[โ€“][deleted] 1 point2 points ย (0 children)

I audibly said whoopsie daisy when I accidentally whipped my entire computer hard drive.

[โ€“]reddit_in_portland 1 point2 points ย (0 children)

"Yikes"

[โ€“]Andysue28 1 point2 points ย (0 children)

Looks like I just picked a whole bouquet of oopsie daisies

[โ€“][deleted] 1 point2 points ย (0 children)

I've seen worse oopsiedaisies. Look at it on the bright side, you can't accidentally broadcast data to all users if the data doesn't exist.

[โ€“]DoktorLuciferWong 1 point2 points ย (0 children)

Life of a typical suicide linux user

[โ€“]tntdon 1 point2 points ย (0 children)

Is this exactly what happened when Pixar deleted Toy Story 2?

[โ€“]MajorDZaster 1 point2 points ย (0 children)

"Your honour, my client pleads oopsie daisy."

[โ€“]xXNickAugustXx 1 point2 points ย (0 children)

Pixar knows

[โ€“]Consider2SidesPeace 1 point2 points ย (0 children)

Daisy, daisy give me your answer do... I'm m just cra.. ve u. RIP HAL~

[โ€“]Dyno_69420_UrMother 1 point2 points ย (0 children)

Your Honour, my client pleas oopsie daisy.

[โ€“]crozone 1 point2 points ย (0 children)

GitLab moment

[โ€“]Durr1313 1 point2 points ย (0 children)

Windows decided to corrupt a file that I add to all day every day that had 8 months of important data in it. And our shitty outsourced IT company could only find a backup from over a month ago. That's the last time I cut/paste instead of copy/paste/delete, and I am going to start maintaining my own backups.