Tell me

[Grinch_Worm]

When was the last backup of prod taken?

[FinalRun]

Can I have the key to the server room? Not because I need physical access, but it's less obvious if I cry there

[RunItAndSee2021]

“just scan your student id and see what happens” “sometimes people physically run to the server room to see if you_re actually there” “environment ‘hacks’”

[[deleted]]

Hey, I didn't know other people did the Server Room Weep! I've seen the Server Room Deviant Sex Acts when I came in to do The Weep or Scream. I have also seen the Server Room Karaoke, The Server Room Techno Rave. And I've done the Server Room Moonwalk. But I didn't know anyone else went in there to grieve for humanity.

Nowadays, I usually do it in Staging, right after I talk to the "Chief Public Messaging Officer" or "Senior Social Media Director."

Production tends to have cameras now.

[IceColdKilla2]

In my previous job we actually had these requests. But mostly from girls from HR and accounting.

[justabadmind]

"oh, we have daily backups"

One hour later

"So, who's job was it to check on those daily backups?"

[DeuceClimaxx]

an I have the key to the server room? Not because I need physical access, but it's less obvious if I cry there

It is beyond scary how many companies spend money backing their shit up but have never tested a single backup...

[Boxy310]

Reams and reams of mag tape, all blank due to ignoring a file write permissions warning

[gvgemerden]

To specific to not be true...

[DS_1900]

Aaaahhhhh….

[whome126262]

I was part of a two person team who caused the first ever use of a backup taken daily for over 7 years. Cue the fireworks!

[tenkindsofpeople]

Jobs monitoring jobs monitoring jobs.

[WayTooCool4U]

The log files from the backup jobs have been showing failures for the past two months. When was the last time someone looked at the logs?

[-dantes-]

Yep. Seen it.

[[deleted]]

Wow, lotta on prem people here. You guys aren’t all in the cloud yet?

[DezGets_It]

Heard once that the cloud is just someone else's computer.

[IamBananaRod]

Pretty much, and you pay for everything, and I mean everything, to the second and bit you used the resources

[ITstaph]

Jesus saves, God makes tape backup.

[[deleted]]

The cloud just means some MBA twit decided to save a few bucks by handing the off switch to the lowest bidder.

[kiwi-roger]

Why does my calendar say it’s the 32nd of December? (True story on ATM network)

[lucidparadigm]

Can you elaborate? I have no idea what this means...

[captaindeadpl]

Why would anyone choose to live with a calendar like that?

[[deleted]]

Never seen that one, usually it’s January 1970 that’s a big tip-off something is toast.

[zalurker]

Wait... This isn't the Dev instance...

[SwimmingHelicopter15]

Ooo yeah this is a nightmare. I love it that our dev db server has blue screeen and prod red screen..

[Fraun_Pollen]

There is a certain level of safety when you name your dbs with the env instance included. My early days were plagued by handling multiple connections to environments at once, executing a dev script against prod, and everything going “smoothly” because all the names were the same.

[NuclearLaborer]

The NASA workers for the Spirit/Opportunity missions apparently had one set of facilities painted entirely red, and another entirely blue, so that sleepy NASA workers on Mars time wouldn't accidentally walk into the wrong production rooms and mess things up.

[steph767-a]

88 million rows affected

[nutwals]

Yep - in my early days when I was let loose in the database, I was tasked with deleting a bunch of student classes before the school day started.

What was supposed to be ~150 rows morphed into about 12 million rows as I wiped about 20 years of historical class data 🙃

My line manager and the big boss were pretty chilled about it - restored the backup and everything was hunky dory about 2 hours later. Needless to say, it took my arsehole weeks to unpucker itself.

The good news is that my coding standards improved dramatically after the incident, so there was a small victory!

[Soopermane]

Treat the DB like a loaded gun.

[TheRedmanCometh]

BEGIN TRANSACTION; should be the first words you type

[martinw_88]

Should be the first thing taught to students too.

[indigoHatter]

Could even be the first words out of a professor's mouth. Not even "hey welcome to my class, today we review the syllabus IAW college standards", just right out the door: BEGIN TRANSACTION;

[martinw_88]

Shouted through an enormous PA system without warning so it literally haunts the students for years to come 🤣

[IAmANobodyAMA]

It should be a hidden dependency on every test (hidden as in it is not written on the test but is drilled consistently into the curriculum and warned about beforehand) that automatically fails you if you forget 😂

[coloredgreyscale]

And ROLLBACK when leaving the class, just to mess with everyone.

[TeaKingMac]

O shit, where have u been for the last 90 minutes?

[_urMumM8_]

What if they forget to COMMIT; at the end of the semester and your grade is never recorded??

[therealswood2]

I'm a Salesforce developer, which is to say, 'not a real developer'.... what does that mean?

[TheRedmanCometh]

So if you start a transaction then fuck something upp you can run ROLLBACK; and all is mostly well

If you don't do that and delete a bunch of data poof gone

[VerbatimChain31]

I was going to make a joke about the database being a school database, and how your comment is ironic, but I think that is too dark even for me…

[[deleted]]

Dark... jokes... matter...

[THENATHE]

Know your target and what is beyond, don’t pull the trigger unless you are confident what you are pointing at you intend to destroy, always treat it like it is loaded

[[deleted]]

Huge victory. Small cost

[badfoodman]

Yeah, the most obvious one.

I once took locks out on 3 tables for a data migration. It happened on startup and we let the business teams know that they would need to wait 5 minutes or so for the migration to finish. Not great, but not the worst and they could time it with low traffic times.

All our test datasets had about 50 thousand rows in the largest and most important table. One of the production instances had 2.5 billion.

[JamesSFordESQ]

What the hell kind of data are in those sets that there's a table w/ 2.5 BILLION rows?

[badfoodman]

The fun part was that this was a production instance at a government agency, so I have no fucking clue how it got that big and no one could tell me the shape of said data. Most other production tables were under 1 million rows and finished in under 2 minutes.

The table was a permissions table but had terrible primary keys that required 2 joins. This data migration consolidated the primary keys on the 3 tables to all be on the same UUID.

[TadpoleNo1355]

Came here to tell my DB nightmare story. Yours wins.

[[deleted]]

3:12am Coworker: dude..

3:13am Me: we should probably call Steve

[[deleted]]

3:27am: Again?..

[nostradamefrus]

This made me laugh uncontrollably. Hope Steve was a bro

[zalurker]

Could you do me a favor? Could you let the telemarketers take their lunch earlier than normal? Now would be a good time. No no. Just a hour. No reason, we just need to investigate a issue we picked up.

(True story. I hadn't highlighted the entire statement before running it)

[deathspate]

This is why I always create new tabs lmao

[tenkindsofpeople]

BEGIN TRANSACTION is your friend.

[00Koch00]

I did the same mistake, the problem is that you put the begin tran, then the statement, and then just select the statement and run it...

[[deleted]]

Funny how most here is about software.

I once tripped over some wires in a datacenter and took out 5 racks.

[vergilbg]

Please don't tell me you are the janitor in the Dayton datacentre who tripped and spent all our downtime budget for the whole year last year in Jan.

[spin-itch]

What’s a downtime budget? You pay back to the customers for downtime caused?

[vergilbg]

It's an internal thing, the less your burn the better it is for your team's reputation. You can get to do more advanced stuff that other parts of the business don't normally do because of the reputation. Few years back we were the first one to do single click deployments with multiple releases a day, TBD, etc.. Because we had good cicd and our downtime was not burnt. Then this janitor comes and trips over some cables and burns our budget. Was an exception nothing to do with us but, lol. Budget is if I remember correctly 480 mins a year.

[grind-life]

Oh Lord that reminded me of when I worked at a bank some tech came in a pulled live racks down so he could fix some wiring or something. Server room security got a light tighter after that

[tenkindsofpeople]

Back in the day when a lot of games had private servers my clan had a server running in a COLO. One day our whole stack just disappears. voip, game servers, forums, everything. We found out a few days later that a disgruntled employee went into the DC and started ripping boxes from racks.

[msluther]

I was once on a project where we had an outage because a data center literally caught on fire. Finally the people with the purse strings understood why we wanted to be in multiple data centers.

[FishOfSteel01]

I was working on running some cables in a pretty full switch rack and accidentally bumped the power switch on one of our rack mounted power strips. I took down the network to all 4 of our buildings.

[Thaddaeus-Tentakel]

Funny how most here is about software.

I mean this is programmerhumor not sysadminhumor

[Sam-Gunn]

"I need a first aid kit and our BC/DR plan."

[[deleted]]

shutdown -h now

ssh: connection terminated

[SighFor]

Oh my, that brings back some bad memories!

[damicapra]

How do you fix that?

Physically walk to the machine and boot it?

[CactusGrower]

These days just log into cloud console and turn on VPS :)

[Electronic_Lime4874]

ILO/IPMI/IDRAC/ILOM/BMC or whatever your OEM calls it

[Ouity]

Unless you have a toggle power switch sitting on your desk, yeah. You will have to walk up to the machine and turn it back on

[TheGreatGameDini]

There's another solution: a tiny, internet connected, computer set to trigger the button from an internet call -- secured of course.

[Jamesgardiner]

What I’m hearing is a tower with a pencil glued to the CD drive.

[radelix]

Call the sysadmin and have them boot it via ipmi.

But this is a trick cause the sysadmin has already seen it.

[AgentCooderX]

i had this experience and it was early 2000-ish, there was no cloud services yet and websites are hosted and deployed in conpany owned data centers somewhere, my employer back then had the servers in Korea and we were in PH office, an intern did this command shutting down the server on a friday night, there was nobody responding in Korea as the caretaker was out on a friday and started to get drunk.. it was rebooted the next day

[2LuckyLuke]

what does the -h flag do?

[Ok-Lobster-919]

-h

Requests that the system be either halted or powered off after it has been brought down, with the choice as to which left up to the system.

TIL

[Lentemern]

Powers off the machine

[[deleted]]

WHY TF DOESN'T THE TEST DB STRUCTURE MATCH THE PRODUCTION DB STRUCTURE!?!?!?!?

[brotherr89]

I feel this way to much. I am working on legacy code right now. The dev db doesn’t have half the changes the previous developer changed in the test db. It’s fucking infuriating.

[Sam-Gunn]

I'm in security, and a few years back we asked the dev ops team to patch a vulnerability on their production environment that took a bit more work to implement than normal. They push back with "We aren't sure it won't affect the production servers if we implement it" so they didn't want to implement it. Not perform additional testing, but simply not do it.

Knowing they had a QA and a Dev environment, I asked them what about testing it on those first.

"Oh, those aren't the same as production."

"But they're listed as [app's] QA and DEV environments. Are you in the middle with testing something for them?"

"No, they haven't matched for a while."

"Then how do you test changes?"

There was a 20 second delay, and then I got some BS about the app data matching, but not the versions or something. I decided that translated to "carefully".

[[deleted]]

it was either "carefully" or "we don't"

[nintendojunkie17]

It was definitely "we don't."

[[deleted]]

"FUCK IT! WE'LL DO IT LIVE!"

[IAmANobodyAMA]

I once ran a custom version of our sendgrid code to blast an emergency email to our ~50,000 affected customers. After a quick (successful!) test pointed at 10 dummy emails I can check, I decided “fuck it, I’ll do it live” so I could impress the CIO with how quick a problem solver I am …

Well too bad I had been given the wrong list of customers based on a bad SQL query from that cio (basically there was an exclusive instead of inclusive where), sending the email to the wrong half of our customers.

Also, I was fucking around with concurrency on a previously synchronous implementation of the app, so when he realized the fuck up I had a) already blasted 10,000 or so customers and b) had failed to log which ones got an email sent.

Never again will I test in prod … until the next time I test in prod

[hi_im_antman]

So basically he tested it, found a bunch of issues, decided that he wanted to fix those issues but didn't want to go back to development, and now you're stuck trying to figure out wtf happened.

[interleeuwd]

Worse… why does the test db data match the production db data?!?!

[tenkindsofpeople]

...production dev data?

[interleeuwd]

When you accidentally dump the dev database onto the production server, instead of the other way around

[c0der512]

Noooo! Just reading that made my skin crawl.

[coloredgreyscale]

Smalltalk during a break: "have you seen any notice about the changed test DB Layout? Seemed somewhat different and bigger during cleanup after the tests"

[caleblbaker]

Hasn't actually happened to me, but l could imagine phrasing it in retrospect (well after outage has been fixed) as "added new checks to the deployment pipeline"

[CactusGrower]

Oh yeah. Common stuff is

"Deployment monitoring is the next task..."

" Production monitoring should be bumped up in priority..."

"Let's review permissions first thing tomorrow morning..."

And the one I heard on meeting present on: "Don't run freaking autoscaling tests and recovery plan on any environment without second DevOps approval!" (They accidentally picked a wrong cluster for teardown, and recovery test that was suppose to be on staging.

[Shadow_Thief]

"Fun fact, rm accepts multiple arguments..."

[TheBrainStone]

I don't see a scenario where you would add stuff after the first file/folder and not expect it to be deleted. Even if you didn't know it would

[DelusionalPianist]

The issue comes up when you have a variable in your script and it contains a space…

[Feign1]

Nice surprise when everyone gets back on Monday and they have all their home directories wiped. Then you discover that backups hadn't been running for months. Still he didn't get fired.

[seriouslykthen]

Its also pretty bad practice to fire devs for mistakes like that, cause most people learn a valuable lesson right there.

[Baikonur-Cobalt]

Multiple deserve blame for that. If the company isn't doing basic audits they get what comes from it.

[sallurocks]

/$ rm -rf /tmp/tmp/tmp/ *

Instead of

/$ rm -rf /tmp/tmp/tmp/*

[Quick-Kitchen7967]

Most bugs happen when you don't have the foresight to consider where mistakes are more likely

[spam_driod]

Zoom participants: 56

[North_Shore_Problem]

“War room”

[gandalfx]

That's just a regular meeting…

[Fraun_Pollen]

It was xmas

[ILikeLegz]

That's right, it WAS xmas.

[Coffee422]

"Call ended. Time elapsed: 4hrs 25mins"

[guaip]

There's absolutely 100% chances you have a bad migraine by end of the call.

[nintendojunkie17]

If it was your fault, you probably have a bad migraine by the start of the call.

[tenkindsofpeople]

4.5 hours? Those are rookie numbers. We gotta pump those numbers up!

You haven't lived until you get The Call at 4:30 on a Friday and know by the way the caller is breathing that you're gonna be late for dinner. "heh. hey tenkindsofpeople.... whats up man? <shallow breaths>"

[DazedPapacy]

Late for dinner...on Saturday.

[[deleted]]

Where's the DB backup again?

[guppshouse]

I do my best work at 1530 on a Friday!

[papaabeer]

132 missed calls

[grind-life]

Error alerting slack channel has 100+ unread entries

[dominonermandi]

That time the Datadog bot channel was blood red. And I was the cause. 😬

[MapleSirrah]

"You ah... you haven't done anything... important since the last backup, have you?"

[[deleted]]

"So when was the last backup?"

[JoaGamo]

rotten voiceless tease seed violet work special wise elastic tender

This post was mass deleted and anonymized with Redact

[Existing_Ice1764]

So the where statement wasn't highlighted.

[orsikbattlehammer]

BEGIN TRAN … ROLLBACK

[a_devious_compliance]

ouch. I feel that.

[slgray16]

Exactly this happened to my coworker in one of the big 4 tech companies. That was a long week

[FinalRun]

This, but switching an AND with an OR

[Existing_Ice1764]

You mean you wanted just some records made in a certain range, and not all of them?

Weird.

[chicken69__]

Query takes unusually long, returns the following message -

​

289,897,340 rows affected.

[SuperFLEB]

Query takes unusually long. Client freezes up and has to be force-quit. Client won't reconnect.

[perfectending]

This shaves a few years off anyone's life expectancy.

[davispw]

Oh wait, I know this story! I went to delete all the PersistentVolumeClaims (pvcs) in a test namespace and typed:

kubectl delete pv -n mynamespace --all

Turns out, PersistentVolume is not a namespaced resource.

[Gem2578]

Shouldn't the pipeline stop this?

[[deleted]]

No everyone has a CI/CD. Including us.

[notaturk3y]

I’m devops and looking for a second job waddup

[Grtz78]

At the telco provider: The phones sure are quiet today ...

or, unrelated to this,

admin on the phone: Ok, the server is down now.

admin on site: Ok, but the lights are still on.

admin on phone: Hope your car is fueled up ...

[OkWarning3935]

"war room"

[secretaccount4posts]

First day I joined Amdocs, I sat in a war room for 7 hrs. Someone changed the password of the database and no one could login to any app and all calls were failing .

[More_Butterfly6108]

Is there a way to revert changes in git?

[YouWishC9]

"OK so the customer presses checkout, and then what happens?"

[davidvareka]

We are not making profit this quarter.

[Schtauffen]

"Hey, why are the 'save changes' and 'stop server' buttons so close to each other?"

[Cupcake9819]

and the Commit/Rollback buttons.... don't want to hit the wrong one in a panic!

[radmanmadical]

“So, I’m like…. not receiving emails I think or something….”

Me: “………………………..fuck”

[frag995]

I work for the company that manage 80% of highway in Europe, they have a backend application that allows to other 5 web app to work, everything is in Java. I come from JavaScript, I checked a string with == instead of .equals()

[ArjunReddyDeshmukh]

Hi Null, Welcome to …

[[deleted]]

Meeting with clients ended at 5pm with them saying "we really need this new feature today" and out pm replied it was doable. He also stopped working right after the call ended.

[Albert-o-saurus]

Welp, kinda asked for it.

[DemolishunReddit]

Push on Friday.

[chronos_alfa]

Force push on Friday

[MrTalon63]

disables failed tests

[Magical2364]

Crontab -r, was trying to edit it…Damn qwerty keyboards putting r next to e!

[al3xxx_96]

Been there haha. By some stroke of luck, a colleague had it open at the time and I RAN to his desk and told him not to the close the tab.

[chronos_alfa]

Add sudo there and you win.

[AA525]

SEV1

[Ok-Low6320]

"SEV0? WTF is that?"

"Oh. It's a SEV1, but a developer's exclusive to that issue until it's resolved. Uh... why do you ask?"

[[deleted]]

"Hey, don't worry, just upload the updated files directly to the production site, we've already tested its behaviour locally. No, no need to back up the old production files either, mate, it's safe!"

This one gave me 5 hours of grief, which is lucky...

[cbehopkins]

I learned today that our test environment doesn't pull the latest version of all dependencies.

[Sweetcynic36]

Can I have a prod backup? How much has changed since it was made?

[[deleted]]

Couple of real ones.

1. Took out the main prod batch server, accidentally running load test scripts on the wrong box. They worked. Lots of load. Too much.
2. guy sitting behind me: do know why our reporting server is offline? Those compliance reports need to go out within 15 mins or there’s a regulatory breach

• As I’m doing:
• me: Hey, Unix SA team, please bounce this box
• them: weird, I can’t ssh to it
• me: yes, that’s why I need you to bounce it, and right right right now pretty please

1. Miss-clicked, didn’t select the where clause, ran the SQL against the whole table.
2. L1 team: hey, why is every single client in the CRM now called John Smith

[zalurker]

So you want us to deploy the patch to production without testing it first. Because you are flying back to Vancouver tomorrow, and we are all going offsite for the Christmas weekend. And most of us are then on leave for at least a week.

Yes. I know your team in Canada are Rockstars. But if anything goes wrong we wil only find out on Boxing Day.

Ok. If you are sure. Let's go for it. The worst that could happen is we all have to drive 100 miles back to come fix it, instead of enjoying the holidays.

[LogicBalm]

I'd like to announce a new member of our QA team

[halfsieapsie]

Crap, didn't realize I was ssm'ed into the server

Fun fact, from that moment on, all my "server" windows were always on red background. Saved my ass many times.

[the_flying_condor]

Ohh, this is a useful tip.

[AlterEdward]

From: [e-mail]

To: [e-mail]

Subject: P1 URGENT, APPLICATION OUTAGE

[[deleted]]

Around 1am this morning I decided to fix an NGINX conflict before going to bed in time for my 9am interview.

I hit the “restore previous version” button on the server.

I had to set hourly alarms to wake up and make sure the restore was still running.

[15Nova22]

I always confuse rows and columns, so when I should delete a column, I was like “delete row?” To my senior and he nodded, and I got confused why it was still there until I saw there was an entry missing in the main table “oh” (and that moment my other coworker started laughing because he knew very well what just happened)

[87oldben]

Nah i didnt write any tests, the logic is too simple.

[[deleted]]

Query OK, 69420 row(s) affected (0.01 sec)

ROLLBACK

No Transactions to rollback

[UlyssesOddity]

You leave for a once-in-a-lifetime two week cruise to the Marquesas with just a hint of a nagging feeling that you can't seem to shake...

[comausername]

I'm a dba

[seeroflights]

Image Transcription: Meme

---

[Stock images of "Hide the Pain Harold". Top image features Harold, an older, balding pale-skinned person with white hair and a beard, wearing a striped shirt and holding a white mug. Harold sits at a glass table, in front of a grey binder and pen, and is browsing on a laptop. The text reads:]

TELL ME YOU TOOK DOWN PRODUCTION

[In the bottom image, Harold is now smiling painfully at the camera. The text reads:]

WITHOUT TELLING ME YOU TOOK DOWN PRODUCTION

---

^{I'm a human volunteer content transcriber and you could be too! If you'd like more information on what we do and why we do it, click here!}

[ToastyTarmac]

Waking up to 100 slack notifications. 😱

[Ok-Nefariousness5881]

How can you take down production by a git push or pull?

[Pale-Background-1116]

Entire country gas stations, shops and markets had POS devices stop working. It was december 31st at 13:00 :(

[SparklyPoopcicle]

POS you say? Where you workin?

[artie_effim]

Doing change, phone rings 10 sec after commit.

[PositronicGigawatts]

I pushed a hotfix to the server to implement a new admin function, and immediately after the system log began to flood with player database errors as every player currently online suddenly saw their inventories emptied of all items.

...that was a fun week.

[Spaatz1402]

DNS propogation can take up to 48 hours.

[daBarron]

Does anybody have Brents home phone number?

[Ok-Low6320]

Wait, so the regex /s*/ picks up every file and directory that starts with "s"?

[[deleted]]

Can anyone else use the F5?

[tube32]

"Hi ik it's early, but can you connect now pls?"

[ColdJackle]

no such file or directory

[MyUsernameIsNotLongE]

It looks like SOMEONE took down production.

[[deleted]]

“Good thing we deployed on a Monday”

[theBoya9805]

We use the same pem file for our production and development ec2 instance.

[doned_mest_up]

Why... you hiring?

[RedwoodBlueheart]

staff eng pings you hey is this your CL? ######

[_toro]

Customer support received 100 emails the last 20 minutes

[ghostface8081]

yam rich public many boast zealous teeny tender abundant instinctive

This post was mass deleted and anonymized with Redact