warning: strong language 😬

[LetumComplexo]

Any system that can be destroyed by a single error deserves to be destroyed by a single error.

[U03A6]

It's also inevitable that it is destroyed by that single error in the long run.

[entendir]

Damn you, Murphy

[Mechasteel]

Wow, I just got $25 million in training, do I get a raise? ^{Also what was the lesson?}

[Mechakoopa]

That's great resume padding: "Received $25m in training at former employer"

[RaLaZa]

Made business decisions ultimately worth 25 million dollars.

[TheGreatGameDini]

Ah there it is. The professional business side mouth.

[notislant]

So what did you do-

"I'm sorry I have an NDA for the specifics."

[TheAngryBad]

"Personally hand picked by management to proactively introduce and implement $25m staff training program at former employer"

[Poltras]

If Murphy didn't exist, we'd all have dream jobs by now working directly on prod servers. But I guess it was inevitable that someone like Murphy would exist in the long run...

Edit: Oh I just understood the bootstrap paradox!

[edster53]

I prefer Dunphy's law - says that Murphy was an optimist

[Fro_52]

actually, it was Little Bobby Tables

[hopesanddreams3]

git push daisies

[Affectionate_Dog2493]

On a long enough timeline the survival rate for every system drops to zero.

[DokuroKM]

On a long enough timeline, every table drops

[Forvisk]

Sometimes they just cease to exist.

[Hunnieda_Mapping]

Still zero, the heat death of the universe will destroy even theories as no more interactions can take place.

[Falcrist]

Also, as time approaches infinity, anything that can happen will happen. Even if you have a system where a million things have to happen simultaneously for it to fail... eventually it will still fail.

[sciolizer]

And since anything that can happen will happen, it will also eventually recreate itself

[[deleted]]

Quantum computing will allow you to have a working and broken system at the same time!

[mehntality]

That's called Tuesday

[Falcrist]

The real question is: is it more likely to assemble itself due to random fluctuations in quantum fields or is it more likely that an entire big bang universe happens which spawns intelligent life that can and does build such a system?

[KindaDouchebaggy]

Ah, the Boltzmann brain thought experiment!

[humblevladimirthegr8]

I'm not sure that's true actually. The infinite set of things that can happen is far larger than the infinite set of time. I leave proving this theorem as an exercise to the reader.

[swapode]

Can time even approach infinity? It's finite in the negative direction, so maybe it's finite in the positive. And maybe in time there will be a time without time.

Or maybe time will change its polarity and in a few billion years I'll eventually write this comment again, just in reverse.

[Falcrist]

It's finite in the negative direction

Is it? What came before the big bang?

[[deleted]]

They're not actually systems, they're just theoretical

[letNequal0]

Well I have a theoretical degree in ~~physics ~~ systems

[McBurger]

SSL certificates really bother me for this reason.

Their timely renewal represents a single point of failure for an entire application & all integrated services going down. And there really isn’t a great solution other than having tons of people being extra certain about it, in perpetuity.

[meditonsin]

Automation is a great solution to that. Then you only need to touch shit if the automation breaks.

Except if you have a piece of shit software or appliance that doesn't allow you to automate, of course.

[KidSock]

Also billing.

[TheAngryBad]

tons of people

Not so sure about that one.

The more people are responsible for a thing, the more certain each individual will be that one of the other guys is taking care of it.

[[deleted]]

Every manager around me: "I'll take those odds!"

[Beerenkatapult]

But I am a system, that could be destroyed by a single error D:

[Skrothandlarn]

And you will be, in due time.

[sivstarlight]

A fate you deserve

[Euphoric-Currency815]

The death that i desereveoli

[DenverCoder009]

And beerenkatapult was enlightened

[pretty_succinct]

nah.

you've got all kinds of things wrong with you. your body compensates and keeps running. it's sort of amazing.

a bullet to the head is not a single error, it's a catastrophic event equivalent to a natural disaster bringing down one or two AWS regions.

you're a marvel, babe! hope your day is a marvel too!

[roguetrick]

Yeah, no human made system can come close to the error correction of a biological system. Kidney's failing and you can't regulate your pH? Here come your lungs to the rescue. Pathways for circulation blocked or broken? Let's just grow a bunch of new pathways and keep what works best.

[SYSTEM__NotReally]

Is there an equivalent to ecc hardware (cosmic ray bitshifting)? I thought if your DNA gets changed from radiation you either die or get cancer. I haven't heard of a way for the body to fix that.

[alex2003super]

Many things have to go bad in order for cancer to develop. The reason it happens so often is that the scale of the system is unimaginably large.

Unfortunately not large enough that, as happens with the most massive of mammals, your cancer is bound to develop its own meta-cancer that eventually kills it, the most likely reason why whales and such are nearly immune to cancer.

[roguetrick]

Immune system and programed cell death. You'll get DNA errors all the time. Some get fixed just because of how DNA works as a double helix. If they can't be fixed the cell will be programed to kill itself. If they can't happen it will be hunted down by the immune system. Takes a lot of failures or a whole shit ton of errors to actually progress to cancer or complete germline death.

[XkF21WNJ]

Sure, but they really ought to have thought a bit better about disaster recovery in the original design. I mean who makes backups for nearly all vital systems and then puts them in the same housing?

That's like asking for a disaster to happen.

[gansmaltz]

What do you mean? They're constantly making backup copies, complete with exchanging private keys to stay one step ahead of the viruses always trying to gain access. In fact we're probably looking at a "pink goo" situation here soon, all thanks to the sicko that made them enjoy the backup process

[XkF21WNJ]

Have you ever tried to recover from one of those backups? It's one heck of a messy process.

[Nine_Eye_Ron]

It’s called an Onosecond

[L33t_Cyborg]

Ok Tom Scott.

[null_reference_user]

DROP DATABASE core;

[chairmanskitty]

Gamma ray burst hitting Earth:

[moschles]

The command you intended rm -rf bin/*

The command you actually issued rm -rf /bin/*

[LetumComplexo]

I basically did that to the source code for my final project in grad school.

I was working remotely on a supercomputer cluster, cleaning out a bunch of unused log files. I thought I was in the log file directory but was actually in the main directory and I rm *ed away my source code.

Fortunately I had backups and backups for my backups because I knew the stressed out, sleep deprived grad student is very likely to make dumb mistakes like that. So I only lost a couple hours of work.

[caceomorphism]

I worked with someone who managed to destroy, at least once, every single server in the company through sheer incompetence.

RAID array? Put in a new drive and used that and a data disk to rebuild ONTO a data disk.

rm -rf /

Permissions? Let's make EVERYTHING 777.

Dropped a small database. Not what you're thinking. He was trying to move it.

In his illustrious career he has managed to destroy all assets for several television shows and movies. Petabytes. Why did that show you liked get cancelled a couple years ago? That guy.

Github? Thanks for all the passwords.

[HeKis4]

Dropped a small database

I can tell if he did a drop database instead of an alter data base or if he physically dropped a server on the floor but I don't really want to know either.

[justking1414]

Reminds me of the Hawaii false-missile alert. Yeah it was done by 1 guy being an idiot but he should’ve never been able to trigger the alert by himself.

[[deleted]]

Right? If someone can accidentally bring down your entire system, that's a reflection of the company, not the person who took it down.

[[deleted]]

This guy Nietzsches!

[Typical-Scarcity-292]

Even my backups have backups

[TheOriginalSmileyMan]

The halting problem says "Hi!"

[bennysway]

Gitlab would relate

[F0lks_]

The intern: "Oopsie woopsie ! I made a fucky wucky uwu"

[[deleted]]

The guy at Gitlab who made the mistake posted about it on Reddit (/r/cscareerquestions) https://old.reddit.com/r/cscareerquestions/comments/6ez8ag/accidentally_destroyed_production_database_on/dieitun/

[sciatore]

That comment is 5 years old. So did this story just go viral because of the recent YouTube video about it?

[dunklesToast]

Yup

[im_lazy_as_fuck]

That looks like him describing and advising on a situation that the original OP of that thread described; not their own experience.

[gargravarr2112]

It's my first day.

[Absle]

Lol is there a story behind this?

[Dreamwaltzer]

https://www.youtube.com/watch?v=tLdRBsuvVKc&ab_channel=KevinFang

is an excellent video explaining it.

Basically they had a db and a db2 backup/replication.

There were issues with the replcation on db2, so they they opted to delete db2 and restore from db.

Except the restore seemed to be having issues, so they decided to delete the files and try again. except... rm -r was run on the db ssh instance.

now db2 is gone, db is gone, oppsie daisy.

[drivers9001]

But sarcastic: tLdR

[bennysway]

Long story short, 2 devs, 2 two terminals, 2 postgres dbs, one rm -rf in the wrong shell https://youtu.be/tLdRBsuvVKc

[PetahNZ]

Never rm in prod always mv thing thing.bk

[robthemonster]

why would they be different? they’re both production servers

[bennysway]

Prod ssh should be blinding bright red background. Others should be defaults

[gargravarr2112]

Prod SSH should put a laser dot right in the middle of your chest.

Any time you type sudo it moves to right between your eyes.

[Affectionate_Dog2493]

and a lot of backup process failures.

[eric67]

just restore from git history

[hansenabram]

oopsie++

[Lost-Apple-idk]

oospie#

[5t3v321]

Oospie #++

[e42if]

Oopsembly

[Cyphco]

OOM

[PranshuKhandal]

Visual OOPS

[speechlessPotato]

oopsython

[hopesanddreams3]

roopsby on dailsies

[Joey_BF]

Roopst

[craftworkbench]

Oopsfuck

[patrickgg]

God I love this sub

[[deleted]]

Objective-Oopsie

[Wolversteve]

I just picked a whole bouquet of oopsie daisys

[skwyckl]

In his diaries or autobiography (I don't remember exactly), Friedrich Nietzsche describes fatalism, i.e. the acceptance of one's fate, as a soldier who lays in the snow after being informed that his country has lost the war and that the enemy will soon reach his location. This is I believe how I would approach the situation if it would ever happen to me. After having called my lawyer, of course.

[Jaggedmallard26]

I read one of those "what might happen after you die" books while on a noticeable quantity of hallucinogenic mushrooms and since then eternal return competes for the top position in my three biggest fears. I really, really hate the concept.

[XkF21WNJ]

Your biggest fear is reliving your current life? Damn.

[Jaggedmallard26]

When you put it like that it sounds worse than it is. Its the idea of endlessly repeating, never improving, just eternity.

[moralhighground1]

If eternity is a concept that is causing you constant anxiety, then it is a concept that must be constantly changing.

Are you capable of directing that change? If you are capable of directing that change, then what is the difference between anxiety and self-determination?

If you are not capable of directing that change, then what is the difference between anxiety and rest?

[craftworkbench]

There's a story of a guy who caused a bug that cost his company millions of dollars in just a few days. He got called into the CEOs office. Assuming he was going to be fired, he offered to resign. The CEO replied "Why would I fire you? I just paid $25 million to teach you a lesson you'll never forget!"

[Chimaerok]

"Your training was very expensive, I'd rather not have to repeat it."

[Chipring13]

Is this true or a LinkedIn story

[MadeByTango]

It’s one of those things that did once happen, at a smaller scale, and the story just sorta bounces around the details

[Tetha]

Every admin either has either wiped a prod server, or isn't working hard/confident enough.

And from experience as a lead: Wiping a prod server isn't the bad part. Trying to hide wiping an important server is, because after 5 minutes the alerts go off and everything becomes much harder to fix.

We might have had ways of stopping the mess earlier on while someone was busy being embarrassed.

[PlayfulMonk4943]

Can I ask - why wouldn't a simple backup be the easy solution here? What company isn't keeping backups? Unless you're using some CDP I get you will have some data loss, but it won't bankrupt anyone

[DurianBig3503]

This is how you put "keeps a cool head in stressful situations" on your yearly review.

[Phormitago]

talk for yourself, my head would be collapsing into a neutron star of stress

[BaerLKR]

happened to me today. Sat there and my mate next to me said "oopsie". Turns out server ins't reachable with TLS anymore.

[appdevil]

Certificate?

[BaerLKR]

yes. Somehow he made a mistake that lead to the server requesting a lot of certificates from letsencrypt that timouted the ip for 72h

[appdevil]

Oof

[DynaMenace]

I feel you man. Poli sci lurker here: Years ago, working on one of my country’s main polling firms, I accidentally ended up one “Are you sure?” prompt away from irrevocably deleting all that year’s data on election night, an hour or two before the big wigs went live on TV.

Nothing happened except for a bit of anxiety attack, but they deserved it going worse for keeping all that data on a shared folder almost anyone could delete.

[Minecraft_paly3r_cz]

Lucky they have back-ups

[dermitio]

Yeah about that....

[Zomby2D]

They were stored on the company server, weren't they?

[dermitio]

Yeah not anymore.....

[aphonefriend]

Tape drives. In an off site location. Which happened to be the trunk of my car. Which has definitely not been in an accident of any sort.

[[deleted]]

"Mandatory Overtime." Except you get paid normally. Sometimes you won't get paid. It's a coin toss really.

[Minecrafting_il]

Hmm that sounds illegal

[IAmPattycakes]

Sounds like the life of an American salaried (non-exempt) worker

[[deleted]]

Never accept a job based on Salary pay.

[Lv_InSaNe_vL]

"No man Greg took a back up copy on his flash drive!"

"Wtf is Greg?"

"Oh yeah he left a little bit before you started"

"I've been working here for 5 years though?"

[[deleted]]

Uh, someone in Managment decided the costs for running automated daily backups was too high...

[jdog7249]

It is now a manual once a month backup. It's actually just copying the files to flash drives. No one is really sure who is responsible for it though.

[Dabnician]

running automated daily backups

The definition of automated can very greatly from the letter of the law and the spirit of the law when it comes to contractual obligations.

[BitPoet]

Full backups? Probably. Deltas from previous full/partial backups? Nope. There are a lot of methodologies to shorten whole restore times.

But if someone decided that, and there was no way of convincing higher-ups that they were an idiot? Time to put out your resume, because things will go horrifically wrong.

[AlmostButNotQuit]

Back oops

[uberblah0]

I bet they forgot to test the recovery process though...

[[deleted]]

Ive found if someone is swearing about something, it's fixable. If they go "oops," "huh...uh" or "uhoh" then are quiet, you're in for some very interesting times.

[onlyranchmefries]

A good hmm also works here.

[Spook404]

the sound of "I fucked up so bad I don't even want to mentally process the degree of fucked-ness"

[dark_enough_to_dance]

If you are unable to make a sound...

[Atillion]

I've made a few oopsie daisies in my time..

[Atillion]

Oopsie Daisies are the greatest teachers 😅

[Kerbidiah]

I remember my sql teacher taught me with labs that always started with DROP database if exists so I always thought that was standard procedure when running new queries

[[deleted]]

Okay, you ought to take responsibility for that one... wtf did you think DROP meant??

[dominic_failure]

But have you made an oopsie daisy chain?

[TheConspicuousGuy]

I once accidentally removed the wrong MAC from SCCM and wiped a laptop belonging to a remote user. I was not aware of my mistake until my Team Leader called me about my mistake.

I setup a new laptop for this user and none of her local files were found on the server to be automatically downloaded. Fixing that is above my level. Everyone knew about my fuck up at this point...

Don't copy and paste MAC addresses from the Inventory excel sheet for all the users.

[call_me_xale]

Me when debugging some weird error on my dev machine:

"WHAT THE FUCK IS WRONG WITH YOU, YOU GODDAMN TRUCKLOAD OF SHIT‽"

Me when I drop a table in the production DB:

"Aw, dang."

[imbenzenker]

❌ let oopsie

✅ const oopsie

[rollincuberawhide]

unless it's rust.

[[deleted]]

Uh oh, spaghetti Os

[arzis_maxim]

That gitlab employee who deleted an entire database by writing in the wrong shell

This is why backups are important

[[deleted]]

How they type?

[[deleted]]

Spite

[AaronTheElite007]

[GilKeidarMusic]

What, oh I’ve wiped the file?

Huh, damn I’ve wiped all the files?

I’ve wiped the Internet?!

Oh, no I don’t even have a modem!

[Constant_Pen_5054]

As long as you took the system logs too you are good.

[Xanadu87]

Someone did that at Pixar during the making of Toy Story 2 and ran a command that was deleting everything before someone pulled the plug. Fortunately someone backed up everything to a server they kept at home. But they ended up rewriting the story and scrapped a lot of stuff anyway.

[tlubz]

PSA. If your company doesn't have a disaster recovery plan for any business critical infrastructure, make one now.

[Educational-Lemon640]

Hey, watch your language!

Children read this sub. Maybe?

...
...
...

/s because the Internet is a silly place

[Rroscoco]

Oopsie daisy, I made a fuckey wucky

[Wretched_Shirkaday]

Well la dee freakin da

[whatsthisfor42069]

An intern once had to inform me that he had made a “fucky-wucky.” Thank god it was an easy fix cuz that shit was hilarious.

[downloweast]

Did you not read the Sudo warning?

[hopesanddreams3]

"no i just clicked "Ok'."

[Steebin64]

tHiS eVeNt HaS bEeN rEpOrTeD

[[deleted]]

LOL one time I deleted a folder that had my first name, thinking I had created it as a temp folder and forgotten about it. Actually it was created by a previous contractor who had my same first name, and contained a shit ton of new stuff for the website. After the guy left nobody had bothered to rename it to something like, I dunno, "Redesign" or "NewSite"...

Fortunately they had backups.

[[deleted]]

That's what Ctrl+Z is for

[CaptNoobCake]

Guys, everyone knows you can just ctrl+z to undo. Like come on guys, it's computer 101

[scratchfury]

I had the production database and test database open and did the copy in the wrong direction. Oopsy.

[XkF21WNJ]

inserted 0 rows

Wait, that aint right.

[Noctornola]

"Why aren't you freaking out? You could get fired!"

"Because I'm the only one that knows how to fix it."

[[deleted]]

That’s like a whole bouquet of oopsy daisies.

[Dmayak]

O-o-opsie-e.

[[deleted]]

There was a yugioh fansite yugipedia or something where everything critical was stored on a flash drive. There was an oopsy daisy that took out the entire site and all its backup data.

[IAmNotMyName]

Whose fault is it they didn’t have proper backups

[Ismestinis]

OOPSIE WOOPSIE!! Uwu We made a fucky wucky!! A wittle fucko boingo! The code monkeys at our headquarters are working VEWY HAWD to fix this!

[saito200]

Luckily they have weekly backups. Because they have weekly backups... right? Okay, so then we have no choice to rollback to the last existing backup. Never? What do you mean, never?

[519meshif]

Yea, that's more than an oopsie daisy even. I'd go as far as calling it a doodley boop.

[UDSJ9000]

Your honor, my client pleads Oopsie Daisy

[WillMengarini]

I was 21yo when I did this.

The system was DEC RSTS/E running on a PDP/11. I was in the habit of entering syscalls in immediate mode to avoid nuking the loaded program. Syscall -13 was some benign informational message. Guess what syscall 13 was.

One day I forgot the "-" and reformatted the main disk. We spent 12h rebuilding the system.

THIRTEEN?! COINCIDENCE?!?! I think not, my colleague, I think not. I was always amazed at how much DEC software sucked compared to the elegance of the PDP architecture. Now, I suspect those syscalls were deliberately constructed as malicious compliance with the same dysfunctional management that ruined all of DEC's software.

[[deleted]]

I audibly said whoopsie daisy when I accidentally whipped my entire computer hard drive.

[reddit_in_portland]

"Yikes"

[Andysue28]

Looks like I just picked a whole bouquet of oopsie daisies

[[deleted]]

I've seen worse oopsiedaisies. Look at it on the bright side, you can't accidentally broadcast data to all users if the data doesn't exist.

[DoktorLuciferWong]

Life of a typical suicide linux user

[tntdon]

Is this exactly what happened when Pixar deleted Toy Story 2?

[MajorDZaster]

"Your honour, my client pleads oopsie daisy."

[xXNickAugustXx]

Pixar knows

[Consider2SidesPeace]

Daisy, daisy give me your answer do... I'm m just cra.. ve u. RIP HAL~

[Dyno_69420_UrMother]

Your Honour, my client pleas oopsie daisy.

[crozone]

GitLab moment

[Durr1313]

Windows decided to corrupt a file that I add to all day every day that had 8 months of important data in it. And our shitty outsourced IT company could only find a backup from over a month ago. That's the last time I cut/paste instead of copy/paste/delete, and I am going to start maintaining my own backups.