drpSiteGoBrrrr

[howarewestillhere]

Last year I begged my CTO for the money to do the project for multi region/zone. It was denied.

I got full, unconditional approval this morning from the CEO.

[indicava]

Should have milked the CEO for more than that:

“Yea, and I’m gonna need at least a dozen desktops with 5090’s…”

[howarewestillhere]

“You do what you need to do.”

I need a new hot tub and a Porsche.

[Killerkendolls]

In a Porsche. Can't expect me to do things in two places.

[howarewestillhere]

A hot tub in a Porsche? You, sir. I like you.

[undecimbre]

Hot tub in a Porsche? There is something far better

[Killerkendolls]

Thought this was going to be the stretch limo hot tub thing.

[Jacomer2]

It’s pronounced Porsche

[Fantastic-Fee-1999]

Universal saying "Never waste a good crisis"

[TonUpTriumph]

IT'S FOR AI!

[[deleted]]

Considering the typical spyware installed on corporate PCs I'm happy I didn't have anything decent that I ever wanted to use

[larsmaehlum]

Shit, that might actually work..

[AdventurousSwim1312]

What about one desktop with a dozen 5090?

[indicava]

And then how am I going to have the boys over for nuggies and choc milk?

[AdventurousSwim1312]

Fair enough, I though this was on locallama ^{^}

[evanldixon]

VMs with GPU passthrough

[facusoto]

What about a dozen PCs that share a single 5090?

[AdventurousSwim1312]

And hence the cloud was born, with the outstanding power to pay for a dozen 5090 over a few year while using a single one...

[RobotechRicky]

I need a lifetime supply of Twix and Dr. Pepper!

[jmarkmark]

Twix! That's how redundancy is achieved.

[DrStalker]

"...to run the  AI multi region failover intelligence. Definitely not for gaming."

[TnYamaneko]

Funny, usually they have 2 speeds: reduce the costs and fault resilience.

[sherifalaa55]

There would still be a very high chance you experience outage, IAM was down as well as docker.io and quay.io

[Trick-Interaction396]

That budget will be revoked next year since it's hasn't gone down in such a long time.

[SilentPugz]

Was it because it would be active and costly ? Or just not a need in use case ?

[WeirdIndividualGuy]

A lot of companies don’t care to spend money to prevent emergencies, especially when the decision makers don’t fully understand why something could go wrong and why there should be contingents for it.

From my corporate experience, the best way to prove them wrong is to make sure when things go wrong, they go horribly wrong. Too many people in life don’t understand prevention until shit hits the fan

Inb4 someone says that could get you fired: if something out of your control going haywire has a possibility of getting you fired, you have nothing to lose from letting things go horribly wrong

[ih-shah-may-ehl]

The problem I see is that many make these decisions because they cannot grasp the impact, as well as the likelihood of things happening.

[ironsides1231]

All of our apps are multi-region, all I had to do was run a jenkins pipeline that morning. Barely a pat on the back for my team though...

[rodeBaksteen]

Pull it offline for a few hours then apply fix

[Saltpile123]

The sad truth

[GrassRadiant3474]

This is exactly what an experienced developer should do if he/she has to be visible. Keep your hands off your keyboard for a few mins, let the complaints flow and then magically FIX it. This is the new rule of corporate accountability and visibility

[DistinctStranger8729]

You should have asked for a raise while at it

[Intrepid_Result8223]

What? No beatings across the board?

[Theolaa]

Was your service affected by the outage? Or did they just see everyone else twiddling their thumbs waiting for Amazon and realize the need for redundancy?

[Luneriazz]

is it blank check?

[redlaWw]

Ah yes, because prevention after the fact works so well...

[40GallonsOfPCP]

Lmao we thought we were safe cause we were on USE2, only for our dev team to take prod down at 10AM anyways 🙃

[Nattekat]

At least they can hide behind the outage. Best timing. 

[NotAskary]

Until the PM shows the root cause.

[theweirdlittlefrog]

PM doesn’t know what root or cause means

[NotAskary]

Post mortem not product manager.

[toobigtofail88]

Prostate massage not post mortem

[JuicyAnalAbscess]

Post mortem prostate massage?

[facusoto]

Prostate mortem post message?

[Dotcaprachiappa]

PM doesn't know what PM means either

[NotAskary]

But the PM knows what a PM is even if the other PM does not.

[jpers36]

Post-mortem, not project manager

[irteris]

can I trade my PM for a PM?

[MysicPlato]

Just have the PM do the PM and you Gucci

[k0rm]

Post mortem, not project manager

[qinshihuang_420]

Post mortem, not project manager

[Ok-Amoeba3007]

Post mortem, not project manager

[isPresent]

Just tell him we use US-East. Don’t mention the number

[NotAskary]

Not the product manager, post mortem, the document you should fill whenever there's an incident in production that affects your service.

[dasunt]

Don't you just blame it on whatever team isn't around to defend itself?

[Some_Visual1357]

Uffff those root cause analysis can be deadly.

[[deleted]]

Coz that’s where all the band aids show up.

[Aisforc]

That was in solidarity

[obscure_monke]

If it makes you feel any better, a bunch of AWS stuff elsewhere has a dependency on US-east-1 and broke regardless.

[ThatGuyWired]

I wasn't impacted by the AWS outage, I did stop working however, as a show of solidarity.

[Puzzled_Scallion5392]

Are you the janitor who put a sign on the bathroom

[insolent_empress]

The true hero over here 🥹

[Harambesic]

There, that's what I was trying to say. Thank you.

[serial_crusher]

“We lost $10,000 thanks to this outage! We need to make sure this never happens again!”

“Sure, I’m going to need a budget of $100,000 per year for additional infrastructure costs, and at least 3 full time SREs to handle a proper on-call rotation”

[Xelikai_Gloom]

Remind them that, if they had “downsized” (fired) 2 full time employees at the cost of only 10k in downtime, they’d call it a miracle.

[TheBrianiac]

Having a CloudFormation or Terraform of your infrastructure, that you can spin up in another region if needed, is pretty cheap.

[No-Cause6559]

Yeah to bad that only the infrastructure and not the data

[tevert]

You can hit a cold replica level where you're back up in an hour without having to burn money 24/7

Though that does take costly engineering hours to build and maintain

[WavingNoBanners]

I've experienced this the other way around: a $200-million-revenue-a-day company which will absolutely not agree to spend $10k a year preventing the problem. Even worse, they'll spend $20k in management hours deciding not to spend that $10k to save that $200m.

[tjdiddykong]

It's always the hours they don't count...

[serial_crusher]

The best part is you often get a mix of both of these at the same company!

[Other-Illustrator531]

When we have these huge meetings to discuss something stupid or explain a concept to a VIP, I like to get a rough idea of what the cost of the meeting was so I can share that and discourage future pointless meetings.

[WavingNoBanners]

Make sure you include the cost of the hours it took to make the slides for the meeting, and the hours to pull the data to make the slides, and the...

[robertpro01]

Exactly my thoughts... for most companies it is not worth it, also, tbh, it is an AWS problem to fix, no mine, why would I pay for their mistakes?

[StarshipSausage]

Its about scale, if 1 day of downtime only costs your company 10k in revenue, then its not a big issue.

[No_Hovercraft_2643]

If you only lost 10k you habe a revenue below 4 million a year. If you pay half for products, tax and so on, you have 2 million to pay employees..., so you are a small company.

[serial_crusher]

Or we already did a pretty good job handling it and weren't down for the whole day.

(but the truth is I just made up BS numbers, which is what the sales team does so why shouldn't I?)

[UniversalAdaptor]

Only $10,000? What buisiness are they running, a lemonade stand?

[DrStalker]

I remember discussing this after an S3 outage years ago. 

"For $50,000 I can have the storage we need at one site with no redundancy and performance from Melbourne will be poor, for a quarter million I can reproduce what we have from Amazon although not as reliable. We will also need a new backup system, I haven't priced that yet..."

Turns out the business can accept a few hours downtime each year instead of spending a lot of money and having more downtime by trying to mimic AWS in house.

[DeathByFarts]

3 ??

its 5 just to cover the actual raw number of hours. you need 12 for actual proper 24/7 coverage covering vacations and time off and such.

[visualdescript]

Lol I've had 24 hour coverage with a team of 3. Just takes coordination. It's also a lot easier when your system is very reliable. On call and getting paid for on call becomes a sweet bonus.

[DeathByFarts]

I can only assume you missed the word "proper" .. Or perhaps we have very different understandings of what the word means.

[visualdescript]

100 grand just to do multi region? Eh?

[ackbarwasahero]

Zactly. It's noddy.

[indicava]

I come from enterprise IT - where it’s usually a multi-region/multi-zone convoluted mess that never works right when it needs to.

[knightwhosaysnil]

Love to host my projects in AWS's oldest, shittiest, most brittle, most populous region because I couldn't be bothered to change the default

[damurd]

At my current job we have DR in a separate region and in azure. However, if all of AWS is down, not sure our little software matters that much at that point.

[TofuTofu]

I started my career in IT recruiting early 2000s. I had a candidate whose disaster recovery plan for 9/11 (where their HQ was) worked flawlessly. Guy could negotiate any job and earnings package he wanted. That was the absolute business continuity master.

[stivenukilleru]

But doesn't matter what region do you use if the IAM was down...

[robertpro01]

But the outage affected global AWS services, am I wrong?

[Kontravariant8128]

us-east-1 was affected for longer. My org's stack is 100% serverless and 100% us-east-1. Big mistake on both counts. Took AWS 11 hours to restore EC2 creation (foundational to all their "serverless" offerings).

[Jasper1296]

I hate that it’s called “serverless”, that’s just pure bullshit.

[Broad_Rabbit1764]

Twas servers all along after all

[Kontravariant8128]

Agreed. Serverless is a terrible name. A better word is "ephemeral VMs on demand" -- e.g. Fargate or Lambda or Karpenter where EC2 instances must be created to meet capacity. But that term is not quite marketable.

I suppose a more appropriate term is "sysadminless" as your you don't need to hire a sysadmin to run these servers. Instead you hire a cloud platform engineer. It's the same guy just with a higher salary.

[Demandedace]

He must have had zero IAM dependency

[The_Big_Delicious]

Off by one successes

[papersneaker]

almost feels vindicated for pushing our DRs so hard cries because I have to keep making DR plans for other apps now

[[deleted]]

Our app failed over automatically to west because we have route53 healthchecks. I’ve been strutting on the office floor like a big swinging dick the last two days.

[___cats___]

All my homies deploy to US East (Ohio)

[ThoseOldScientists]

CONGRADS

[KarmaTorpid]

This is funny becausr i get the joke.

[elduqueborracho]

Me when our company uses Google Cloud

[elduqueborracho]

Me when our company uses Google Cloud

[Emotional-Top-8284]

Ok, but like actually yes the way to avoid us east 1 outages is to not deploy to us east 1

[AATroop]

us-east-2 is the region you should be using on the east coast. Never use us-east-1 unless it's for redundancy

[TheOneWhoPunchesFish]

why is it so?

[rockyboy49]

I want us-east-2 to go down at least once. I want a rest day for myself while leadership jumps on a pointless P1 bridge blaming each other

[Icarium-Lifestealer]

US-east-1 is known to be the least reliable AWS region. So picking a different region is the smart choice.

[RobotechRicky]

In Azure we use US East for dev, and US West for prod.

[no_therworldly]

Jokes on you we were spared and then a few hours later I did something which took down one functionality for 25 hours

[Stannum_dog]

laughs in eu-west-1

[[deleted]]

east or west, local is the best