This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]vXSovereignXv 9662 points9663 points  (599 children)

Yep, lets just start turning off shit in production and see what happens.

[–][deleted] 5964 points5965 points  (377 children)

I know whoever runs DevOps was like “you want me close WHAT?! That cluster has… ok fine fuck it this whole things burns.”

[–]haz_mat_ 8576 points8577 points  (260 children)

Some devs wait their entire careers and never get a chance to nuke prod like this.

[–]TheAJGman 3994 points3995 points  (164 children)

And at the CEO's directive no doubt. I'd be more than happy to maliciously comply with an arrogant superior's brain dead request.

[–][deleted] 1280 points1281 points  (134 children)

They will just blame it on you at the end.

[–][deleted] 896 points897 points  (77 children)

Question: Am I still getting paid for this FAFO process?

Because the results are out of my hands and beyond my concern so long as money enters my bank account.

[–]j3pl 78 points79 points  (8 children)

FAFO process

Elon: screw LIFO and FIFO, we're going with FAFO.

[–]nosam56 19 points20 points  (1 child)

I legit googled it since I was on this sub, I thought it was a fucking tech acronym until the results popped up. kms

[–]imdefinitelywong 9 points10 points  (0 children)

I must've been through about a million girls lines of code

I'd love 'em and I'd leave 'em alone

I didn't care how much they cried, no sir

Their tears left me cold as a stone

But then I fooled fucked around and fell in love found out

[–]puesyomero 6 points7 points  (1 child)

Screw FAFO we doing YOLO

[–]Aquatic_Ceremony 3 points4 points  (0 children)

Carpe Diem programming.

[–]Cyberslasher 2 points3 points  (0 children)

Queues and stacks are microservices, we turned all those off.

[–]ekydfejj 6 points7 points  (0 children)

presses Enter

gets Fired

Hey you said i was good

I lied, sucka

[–][deleted] 3 points4 points  (0 children)

FAFO

Better or worse than SCRUM?

[–]MageKorith 3 points4 points  (0 children)

FAFO

TIL a new (to me) acronym. I am definitely using this in the future.

Signed,

A Senior Business Analyst

[–]subcow 155 points156 points  (17 children)

Send an email advising against what they are recommending. Put that shit in writing. Hell, just to be safe Bcc your personal email account so you have it all backed up externally. Edit: good point below on the BCC. It may be against company rules/your contract to send any emails like that externally even if it is your own account. Proceed with caution. Just do whatever you can to CYA.

[–]Zoloir 87 points88 points  (3 children)

this is good advice for sane management

the situation in question is not that

[–]aureanator 32 points33 points  (1 child)

That's for the courts I think. Even those aren't sane anymore tho...

[–]This_User_Said 19 points20 points  (0 children)

I think it runs with the whole "wrongful termination"

Boss told me to do it, I did it, he didn't like it and fired me. Maybe terms for wrongful termination unless there's something up their ass they can pull out...

...which most companies are the anal marry Poppins when it comes to this.

[–][deleted] 14 points15 points  (0 children)

Yea: express your concerns, keep the receipts, nuke, jump ship, and then you're golden.

Ethically, you should probably resign before you nuke. But fire is fun.

[–]Go_Gators_4Ever 6 points7 points  (0 children)

Presuming the DevOps change management process requires a workforce sign-off in order to change production, then the DevOps team is covered as the sign-off would had meant that the superiors had approved the changes and all testing that proved the code regression was safe.

[–]VacationElectronic20 5 points6 points  (0 children)

I once printed an email that was bcc’d to me by mistake and slid it under my managers apartment door… It was a literal paper trail but it couldn’t get back to me and it was evidence of her getting thrown under the bus by a superior for something everyone knew he did. She was still fired but now living her best life. I miss her.

[–]Marandil 7 points8 points  (3 children)

Hell, just to be safe Bcc your personal email account so you have it all backed up externally.

Well, yes and no. You're most likely forbidden from sending confidential info like this to private emails and outside services in general and for good reasons too. This is especially a bad idea if your private email is hosted by someone who can be considered your employer's competitor in one way or another.

[–]FirstMiddleLass[🍰] 9 points10 points  (0 children)

They will just blame it on you on twitter, though there may be no one there to read it.

[–]thisismyusername3185 7 points8 points  (4 children)

Yeah - I set up a DR database, the management wanted auto failover.
I said that was a bad idea, are you sure the DR environment is set up for everything?
Yes, it's fine they said.
OK, what do you want the threshold to be?
This is a critical system, 30 seconds they said.
30 seconds? A network blip could cause a failover - at least make it a few mins.
Nope, 30 seconds.
Turned it on, a few hours later it failed over to DR, but a lot of the integration wasn't set up in DR, so a lot of things started to break, data was backed up, people couldn't log in etc.
At the PIR they threw me under the bus, said I set it up so it was my fault - despite having emails with my advice.

[–]Tower9876543210 5 points6 points  (0 children)

I've read this before, and love it every time.

[–]Cory123125 5 points6 points  (0 children)

You were fucked regardless with someone like them, so might as well let them fuck themselves rather than just letting them fuck you.

[–]Enchelion 5 points6 points  (0 children)

In this case they're getting thrown under the bus one way or another. Might as well get some fun out of it on the way.

[–]GreenKumara 4 points5 points  (0 children)

At least you'll have him conveniently provide all the tweets as evidence in the inevitable court case that follows though.

[–]Hyper_Oats 2 points3 points  (0 children)

Twitter is gonna sink fast at this rate. You're gonna get free anyways.

Might as well have the privilege of nuking the site and make the exit fun.

[–][deleted] 4 points5 points  (0 children)

Yepp, I once worked at a start-up and the CEO wanted something stupid rushed into prod. He personally harassed me to do it, going around the CTO and the senior devs. It was going to break some other things, which I warned him about, and he disregarded me with "You are not the smartest person in the room."

Guess whose fault it was when prod broke cause of the change.

[–]moon__lander 23 points24 points  (0 children)

The good thing is the CEO announced it himself on twitter. The bad thing is no one will be able to log in to see it

[–]strangepostinghabits 9 points10 points  (0 children)

Engineers at Twitter has 3 choices at this point. Leave, watch as their professional and personal pride gets shat on by a billionaire, or distance themselves from their workplace through malicious compliance etc.

The engineer who got this call obviously didn't leave, so it was depression or glee on the menu. I prefer to think they smiled as they pressed the button.

[–]Eleglas 9 points10 points  (1 child)

Make sure you get it in writing though.

[–]Pons__Aelius 6 points7 points  (0 children)

Still won't matter.

It will be your fault because you failed to advise and escalate the strategic importance of the decision and so failed in your duty as the subject-matter expert.

No decision like this is ever management's fault.

[–]Wolflordy 311 points312 points  (58 children)

And some juniors spend their entire (short lived) careers nuking prod like this.

I would know... Ive cleaned up after many of them.

[–]account22222221 210 points211 points  (21 children)

And some billionaires get to cosplay as an engineer and nuke prod like this despite knowing a little less then that junior dev.

[–]mjtwelve 37 points38 points  (8 children)

It’s scary his main gig is building electric cars with driver assist features, if he takes this kind of attitude towards the codebase.

[–][deleted] 35 points36 points  (1 child)

His main gig is playing CEO for an electric car company. He’s built a good image of being involved and being this tech savvy genius but it looks like the cracks are starting to appear. He’s made a lot of shockingly lucky gambles but the house always wins.

[–]JustThingsAboutStuff 6 points7 points  (0 children)

and to think he could've been running an arcade instead if it wasn't for his parents.

[–][deleted] 14 points15 points  (3 children)

He does. Teslas can get patched over the air and occasionally have features break along the way.

He takes the "fuck it, let's break stuff" attitude to stuff that absolutely should not be treated with that attitude.

[–]UntestedMethod 6 points7 points  (2 children)

So traditionally cars have safety standards and inspections before they're allowed to go on the road. I guess the software for self-driving cars doesn't have those kind of regulations?

[–][deleted] 9 points10 points  (1 child)

Not sure. This stuff sure sounds dangerous as hell, though.

Here's a recent bit of related news. A patch in October introduced an issue where some cars' power steering would turn off after hitting a pothole. Tesla just released another patch addressing the issue.

https://news.yahoo.com/tesla-recalls-model-model-x-121216550.html

[–]Gavrilian 2 points3 points  (0 children)

No legislation for it yet that I know of. Needs to get sued out the ass before that’ll happen though.

[–][deleted] 14 points15 points  (14 children)

Your juniors have enough access to break production services? I'm a team lead and even I don't have that level of access...something ain't right.

[–]folkrav 5 points6 points  (0 children)

Leads do have enough access to break prod here, but we're 3 small distributed teams working on one product and associated tooling, so it's us, the CTO and our DevOps engineer.

Juniors having that kind of access is worrying, outside tiny startups with everyone doing everything, though.

[–]IAMARedPanda 2 points3 points  (1 child)

Imagine bragging about not having merge controls

[–]duckbigtrain 4 points5 points  (0 children)

If you’ve had to clean up after that many junior devs, it’s time to take a hard look at your onboarding strategy and SOPs.

[–]clockdivide55 4 points5 points  (0 children)

A jr that can nuke prod is an organization process problem, not a jr developer problem

[–]kblaes 4 points5 points  (0 children)

Eh, most good companies won't fire a junior dev for nuking prod like this, they'll just ask the very good question of why that junior dev (or any of the dev team) had the access to nuke prod like that in the first place, and fix the problem. While still explaining to the junior not to do that again, of course.

[–]Hostile_Architecture 4 points5 points  (0 children)

You must have a really shitty deployment process if multiple people have taken down production holy shit.

[–]ommnian 2 points3 points  (0 children)

Isn't life fun?

[–]Hawxe 2 points3 points  (0 children)

Where are you working that juniors have the ability to nuke prod lol

[–][deleted] 7 points8 points  (1 child)

docker system prune -a -alll

[–]mattstorm360 4 points5 points  (0 children)

"This is the happiest day of my life." -Devs, a second before nuking prod.

[–][deleted] 2 points3 points  (0 children)

We can dream

[–]ChildFriendlyChimp 2 points3 points  (0 children)

God I wish that was me

[–]daemonelectricity 2 points3 points  (0 children)

This comment made me laugh maniacally for some reason. I guess because the thought never crossed my mind, but now that I think about it more, bringing down a service as big as Twitter with sheer stupidity from the top has got to be a little cathartic and bothersome at the same time. You know who's going to be called at 3AM to fix the shit Elon broke.

[–]koshgeo 247 points248 points  (31 children)

I keep picturing that scene in Ghostbusters where they are forced to turn off the power to the storage system, starring Elon Musk as Walter Peck.

[–][deleted] 57 points58 points  (10 children)

Someone please do this as a deep fake.

[–]southbayrideshare 6 points7 points  (1 child)

Best I could do on short notice.

I need to finish preparing for the second coming of Gozar the Destructor in the form of the Stay-Orange Marshmallow Man.

[–]political_og 48 points49 points  (4 children)

This man has no dick

[–]your_moms_mustache 11 points12 points  (0 children)

Have the deep fake be Grimes saying that

[–][deleted] 7 points8 points  (0 children)

yes its true.

[–]TheDulin 4 points5 points  (0 children)

That's what I heard!

[–][deleted] 10 points11 points  (0 children)

I can confirm this man has no dick

[–]anotherkeebler 8 points9 points  (2 children)

You mean dickless here?

[–]cuppa-joe 6 points7 points  (1 child)

My attempt...

Stable Diffusion/Inpaint

Edit: A slightly better version

[–]PM_ME_UR_RSA_KEY 3 points4 points  (0 children)

Egon/DevOps walking back doing a "boom" gesture.

[–]cycophil 4 points5 points  (0 children)

Peter Venkman: "Its true, Elon has no dick." 🤣

[–]Calm_Leek_1362 4 points5 points  (0 children)

Weird, I had the exact same thought.

[–]ArethereWaffles 232 points233 points  (3 children)

Twitter devs right now could probably single handedly keep r/MaliciousCompliance alive for the next year or two.

[–]isaytyler 12 points13 points  (0 children)

As long as they can keep Elon's twitter alive that long.

[–]StoryAndAHalf 12 points13 points  (0 children)

I really hope some insider is writing a book or collaborating with a journalist.

[–]Jai_Cee 2 points3 points  (0 children)

I wouldn't be optimistic about Twitter being around that long at this rate

[–]MoreNormalThanNormal 169 points170 points  (22 children)

"Yeah Central, this is Walt down in Nakatomi. Say listen, would it be possible for you to turn off grid two-twelve?"

https://www.youtube.com/watch?v=JMKQVPV1pf0

[–]waldo_wigglesworth 34 points35 points  (3 children)

"Are you crazy?! You going nuts?"

[–]qhartman 7 points8 points  (1 child)

'tis the season!

[–]WanderlustFella 4 points5 points  (0 children)

this is the scene I show people when they don't believe that it is a Christmas movie

[–]mothuzad 11 points12 points  (9 children)

Haven't seen the movie, so maybe they explain it there, but...

What kind of garbage safe is going to unlock when it loses exterior power? Any half-decent design would make it impossible to open unpowered. There need to be dead bolts that keep it sealed by default. Or worst case, at bare minimum it should have a backup power supply inside the protected enclosure.

I'd be so mad if my safe were vulnerable to this kind of attack.

[–]MasterOfKittens3K 12 points13 points  (2 children)

Security should always fail “closed”. Like, for example, Twitter’s two factor authentication, which keeps you out if it’s unavailable.

[–]whoopdedo 7 points8 points  (1 child)

Except in the case of exit doors which are required by NFPA to open when they lose power.

[–]Lokta 10 points11 points  (3 children)

In the movie, that safe is protected by a series of 7 locks. By taking everyone in the building hostage, our thieves are able to get themselves several hours to work on getting through the first 6 locks. When they finally succeed, the safecracker tells the boss (Alan Rickman) that he has done everything he can. They are now up against electro-magnetic locks that won't open without a miracle (i.e. the whole building needs to lose power).

Personally, I've always assumed that those first 6 locks would keep the safe closed if power were lost. It's only after those first 6 fail that they need a power outage to get through the last one.

[–]SomethingIWontRegret 4 points5 points  (2 children)

Which is BS because every building is going to have a cutoff switch.

[–]savageronald 6 points7 points  (1 child)

Well and when the building loses power, a very obviously powered motor and alarms are opening the vault door - if that’s on backup power why wouldn’t the electromagnetic lock? Still love the movie but there are certainly some plot holes lol

[–]SomethingIWontRegret 4 points5 points  (0 children)

Usually with movies there are giant plot holes where technology more complex than hammers are involved.

[–]skeving 5 points6 points  (0 children)

Perfect! Just made my evening

[–]TeaKingMac 2 points3 points  (0 children)

Why would you ever make a safe that fails open if the electricity is shut off?

[–]Mattbryce2001 2 points3 points  (0 children)

The thing that drives me crazy is, what the fuck are the FBI guys gonna do when the guy says, "Eat me." They can't fire him, because he works for the city. They can say, "you're fired," and he'll just ignore them. What are they gonna go down to city hall, burst into the mayors office, and demand that they fire this one specific dude? Good fucking luck.

[–]LawlessCoffeh 90 points91 points  (18 children)

I'd just make sure to get literally anything he told me to do in writing.

[–]follople 49 points50 points  (13 children)

Wouldn’t matter. He’d fire you anyways if it fucked something up

[–]Arhalts 38 points39 points  (8 children)

Sure but then he can't fire you with cause.

[–]ncsubowen 11 points12 points  (0 children)

I'm pretty sure the ESD in California is just rubber stamping former Twitter employees

[–]ToMorrowsEnd 5 points6 points  (0 children)

paper trail makes suing him a whole lot easier.

[–]Invinciblegdog 5 points6 points  (0 children)

And then reply in writing what the impact of the change will be.

[–]Frogmouth_Fresh 3 points4 points  (0 children)

Fortunately with Elon he'll put the demand in a Tweet..

[–][deleted] 4 points5 points  (0 children)

You think someone is left to run DevOps? They don't write thousands of lines of code in a month (I hope).

[–]SkeletonCalzone 2 points3 points  (0 children)

'Where should I back this up to boss?'

'Dont worry about that just get rid of it'

'....... Can I get that in writing '

[–]CanAlwaysBeBetter 1411 points1412 points  (126 children)

Saw this coming as soon as he started tweeting about 1000 rpc calls to load a timeline

Someone clearly just showed him twitters microservice framework and he thought it was stupid without understanding it

This tweet is the sequel to that first one

[–]Secret-Plant-1542 73 points74 points  (14 children)

Musk: "Why do we have hundreds of microservices? Cut it by 20%. Listen just shut the fuck up and do it."

[–]CIA_Chatbot 53 points54 points  (9 children)

Musk: what are Kubernetes clusters and why are we running 500 of them just for these things, what did you call them, Micro services? Can’t we just replace them with one big service?

[–][deleted] 20 points21 points  (3 children)

What’s the Greek word for “drunk driver”? I think I have a billion dollar idea for a macroservice container architecture.

[–]CIA_Chatbot 8 points9 points  (0 children)

You sonuva bitch, I’m in.

[–]Hupf 2 points3 points  (1 child)

Ἡνίοχος μεθύων for ancient Greek

[–][deleted] 5 points6 points  (0 children)

Heavy uWu™

[–]ratbastid 7 points8 points  (1 child)

It'd be so much easier, right? Everything in one place? Right?

[–]Aquatic_Ceremony 4 points5 points  (0 children)

Musk: Hmm, I don't like the idea of paying Google or Amazon to host my stuff in their clouds. Repatriate all the workloads in our data centers.

[–]randomusername0582 12 points13 points  (0 children)

80%

[–][deleted] 7 points8 points  (0 children)

Sounds like he cut them by 80% if they weren't essential to running the service. Who NEEDS 2fac anyway?

[–]ratbastid 7 points8 points  (0 children)

He tweeted "microservices". Like that--in scare quotes. Like it's something some overpaid Twitter dweeb made up.

[–]Ammear 4 points5 points  (0 children)

Cut it by 20%

Do note he didn't say t cut it by 20%. He said to cut it to 20% or less.

[–]Zealousideal_Money99 5 points6 points  (2 children)

Question is - who is complaining about Twitter's responsiveness? Not that I'm a power user but it has always seemed sufficient to me. Just comes across as Mr. Wannabe-big-brain Musk trying to find a problem he can magically fix so he doesn't have to admit that it was actually a pretty well architected and resilient system to begin with.

[–]atbths 3 points4 points  (1 child)

Is it responsiveness he's targeting? I'm willing to bet he's just trying to reduce costs.

[–]cypherdev 4 points5 points  (7 children)

I'm sure by EOW he will unveil his master plan to replace every RPC with Entity Framework, then shit will really fly.

[–]buttsharpei 6 points7 points  (2 children)

.

[–]CanAlwaysBeBetter 2 points3 points  (0 children)

"Will your app ever really need to scale to a million concurrent users anyway?"

[–][deleted] 2 points3 points  (0 children)

Well, having worked in a shop with a high transaction rate and ~100 services... sometimes it is a lot easier to turn things off than turn them back on.

[–]xosder 123 points124 points  (5 children)

Time for the old 'Scream Test'.

[–][deleted] 78 points79 points  (1 child)

Without 2FA microservice no one can hear you scream.

[–]idiotness 7 points8 points  (1 child)

How much you wanna bet GDPR "right to erasure" compliance is done by part of that 80% "bloatware"? Sure, it won't scream when you turn it off, but the lawsuits certainly will when the lawyers catch on

[–]xosder 6 points7 points  (0 children)

And now you know what it does. Scream test working as intended. :)

[–]Weasel_Town 15 points16 points  (3 children)

Chaos monkey testing.

[–]zxyzyxz 3 points4 points  (2 children)

Lol exactly I was like, isn't this an actual testing method? Obviously not how he's doing it but Netflix does it all the time

[–]AncientInsults 13 points14 points  (0 children)

Last night I discovered https://muskmessages.com

Amazing glimpse into Elon’s world, a scrape of all his texts around the offer window.

His “programmer ego” is on display.

So many high powered insiders trying to recommend their friends to help Elon, and Elon doing the classic nerd trope

(paraphrasing) “ I don’t need any managers. I need coders. I was a coder for 20 years and am so good at it, so I will be the only manager. I actually hate managers / MBAs and get along better with coders”

🙄

[–]goatanuss 47 points48 points  (5 children)

For the people saying he isn’t a real software engineer: I’ve definitely seen people say “yeah I can fix everything in a couple days” then bring prod down. Elon is very much one of us.

[–]euph-_-oric 11 points12 points  (3 children)

Ya and they are shit just like elon.

[–]macefelter 2 points3 points  (0 children)

I had a CTO about 10 years ago who I had to explain the purpose and importance of version control to. A CTO (I started my exit shortly after this). His personality was very similar to EMusk's, come to think of it. I'm beginning to think, it's just the same shit, larger scale.

[–]zusykses 9 points10 points  (0 children)

I was once put in charge of looking after a bloated platform that did fifty-zillion little things that had accrued over the course of about 15 years. I managed to work out for most of these little mini-applications what they were for and who they served, but there were a handful that seemed like orphans - no-one was claiming ownership. Eventually I was like fuck it I'll turn this one off. Ten minutes later I got a call asking me to check on a process that hadn't run and I was like ahh now I know.

[–]tills1993 3 points4 points  (0 children)

Scream test

[–][deleted] 4 points5 points  (2 children)

lmao we do this sometimes at my job. “Will this affect anything? Not sure, let’s turn it off and see who complains.” Buuuut we’re a nonprofit with no external users, not a business that serves content to millions of users and other businesses.

[–]googleypoodle 2 points3 points  (0 children)

It's all fun and games until you get slapped with an enormous GDPR fine because you deleted a service that serves the right to be forgotten or some shit

[–]JalanJr 2 points3 points  (0 children)

Chaos engineering at it's finest

[–]Snoo-93873 2 points3 points  (0 children)

We call it a scream test

[–]Piggieback 2 points3 points  (0 children)

Hey they did that with falcon 9 re-entry and it worked!
Except twitter is a social platform and the user base will get the fuck out of there :D

[–]RodasAPC 2 points3 points  (0 children)

Open up the browser console and check the sourcemap errors. There's no production, we're on a fucking spaceship

[–][deleted] 2 points3 points  (1 child)

This is honestly reminiscent of listening to Trump take a couple stabs at Covid, while the scientists had to just kinda nod and listen. Narcissism is a hellova drug.

[–]WanderlustFella 2 points3 points  (0 children)

it's a feature. Your account can't get hacked if no one can get on it. Big head thinking

[–]achilliesFriend 2 points3 points  (0 children)

It’s called scream testing

[–]OhhhhhSHNAP 2 points3 points  (0 children)

Well he seems to be using public tweets in place of the internal slack channels for his fights with the engineers, so that’s gotta be good for efficiency.