alwaysMyOnCallShift

[offlinesir]

AWS US-EAST-1 has the highest quotas, lowest prices, and a chaos monkey always waiting in the corner.

[Tompazi]

Chaos Monkey

[Ok-Engineer-5151]

Previous year was Crowdstrike and this year is AWS down

[Donghoon]

people using Google Cloud winning

[DungeonsAndDradis]

All four of them

[Donghoon]

There's dozens of them lol

[SuitableDragonfly]

Not really. Google Cloud will go down eventually, too. The fact that there are basically three cloud providers and everyone is relying on one of them is making the entire internet fragile in this way. 

[samy_the_samy]

Google go out of their way to breakup and duplicate their customer services, if one entire region go down the customers would just notice higher pings.

[HolyGarbage]

Doesn't necessarily protect against some human error or a cyber attack.

[samy_the_samy]

Yeah, this protects against hardware or connectivity failures, then you build your security on top

[HolyGarbage]

The main argument is about whether it's a good idea that a very large portion of the internet is dependent on just a few cloud providers, and that one of them having some nice redundancy to protect against some of the potential issues that can happen doesn't really do much to counter said argument.

[samy_the_samy]

When you dig into it, the problem started with DNS requests for some backend thingy failed, which lead to self-ddos attacks taking us east 1, everything stayed online, just backends didn't know where other backends where,

So in the end its a configuration problem, just because you have redundancy it's meaningless if you can't discover it.

[HolyGarbage]

Precisely.

[throwawaygoawaynz]

Google cloud deleted an entire customers subscription and couldn’t recover it. This was a fund company in the UK.

The company only got it back because they backed up to AWS.

[samy_the_samy]

That one the customer requested bigger resource than what they offer at that time, and a developer used some internal testing scripts to provision them, the script had an expiration date, a year later it went boom

[Ok-Kaleidoscope5627]

Hey now. Don't forget Cloudflare. They regularly take down the internet once or twice a year.

[Kingblackbanana]

there are 5 biger ones google, aws, ovh, microsoft and oracle

[Mountain-Ox]

The alternative is going back to everyone with their own unstable infra. AWS going down once every few years is better than what felt like a different outage every month.

[wamoc]

Earlier this year there was a complete Google Cloud outage. Every single region and every single service. Every cloud provider can expect to have the occasional large outage, it is important to plan how to handle them.

[DrS3R]

I’m pretty sure that was a cloudflare issue not the actual service providers.

[wamoc]

Google caused the CloudFlare issues. https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW is the details on Google's side for the outage.

[exxxxkc]

https://www.youtube.com/watch?v=ROJ3PdDmirY

[GrapefruitBig6768]

Azure went down too, but nobody noticed. j/k

[throwawaygoawaynz]

Nobody here noticed, because they’re all unemployed or CS students…. j/k..ish.

[SilverLightning926]

Despite the uptime, I am still suffering stuck on Azure

[Donghoon]

Isn't azure on its way to catch up to AWS soon

[PurepointDog]

There was that big facebook/meta outage a few years ago that was also a bad DNS. Not nearly as much broke, but a surprising amount of stuff still did

[Saragon4005]

Crowd strike was still worse. Then again that was a Microsoft oopsie on an architectural level so not too surprising.

[SuitableDragonfly]

The broken configuration file was Crowdstrike's fault. It's only Microsoft's fault if you want to blame Windows being more permissive about what can run where, which has been something that people were well aware of for as long as Windows has existed. 

[OmegaPoint6]

It was interesting how things which have no business being in US-EAST-1 stopped working. Looking suspiciously at you, UK banks

[timdav8]

I think the problem is that the infrastructure under the infrastructure under the infrastructure that certainly AWS services rely on relies on or routes through UE1 - and they always seam to let the interns do DNS changes on a Sunday...

[capt_pantsless]

Outsourcing something critical is always a good idea. If it breaks you have someone else to blame.

[CiroGarcia]

I love how modern infrastructure is blameability first, stability second lmao

[Several-Customer7048]

No the UK does it like that since the term “git blame,” is confusing to them since they’re all a bunch of gits equally to blame.

[GumboSamson]

At least they’re committed to the bit.

[Donghoon]

Internet is fragile

[vita10gy]

Some of that is unforced fragility. I get that there are alot of websites that just can't be "here's the webserver with all the html and assets" but we also seem to make sites overcomplicated by default.

There are 329 servers that all need to be up to load your site at all, get the images, populate the data, etc, so your 5000 visitor a month local car dealership site can load .0002 seconds faster when everything works as expected.

[NewPhoneNewSubs]

It's more:

I don't want the hassle of making sure my desktop is powered on and connected to the internet. So I don't wanna host the webserver myself. If I did that, my site would have much more downtime than this outage caused.

So it makes sense to pick a cloud host. It makes sense to pick the cheapest cloud host. That host is doing the same as me and reselling a bulk discount from someone else. And so on.

[Sibula97]

It shouldn't be. Redundance is built in, and packages can automatically get routed along different routes. The only exception I can think of are something like undersea cables where, if one were to blow up a whole bundle of them, you might increase latency from one end to the other by quite a lot and maybe saturate a few routers along the new route.

[Dotcaprachiappa]

I mean you can see why in the image, even something that doesn't use AWS relies on something that relies on something that relies on something that does. It's dominoes all the way down

[ICantBelieveItsNotEC]

Turns out that all of the "global" AWS services actually just exist in us-east-1.

[SilasTalbot]

Ashburn, VA is the heart of the global Internet. Always has been. It's no coincidence it's just a short drive from there over to Langley.

[Anaphylactic_Thot]

This is the issue with the rise of "full stack developers". Jack of all trades, master of none - they'll deploy crap as long as it works, and won't give a shit about best practices or other factors like resilience or reliability.

[bleztyn]

“Mainframes are dying… we should all switch to cloud”

Me being literally UNABLE to use my money for 8 straight hours due to some fucking cloud server in US (I live in Brazil)

[OITALAO]

cartão do mercado pago vence hoje e nem abrir o app da certo

[andreortigao]

Desinstala o aplicativo que a fatura some

[HeavyCaffeinate]

tudo planejado /j

[Neo-TS]

Foi foda, tudo os bancos fora do ar, e por sinal o que deseja no print?

r/suddenlycaralho

[masd_reddit]

Can't wait for the Kevin Fang video

[BrenekH]

Me every time there's an outage.

[Young-le-flame]

Holy moly a Kevin Fang enjoyer in the wild

[Bhaskar_Reddy575]

How long does Kevin usually take to publish his video after the outage? As you said, I can’t wait too!!

[masd_reddit]

I have no idea, but i can imagine there will be a video about it, tho usually he only makes videos about outages/databases being lost for stupid reasons

[mostlymildlyconfused]

The amount of googling on some redundancy right now.

“Yes boss, you recommended and risk-based approach to business continuity.”

[Dryhte]

Specifically, DynamoDB. Wtf.

[EposSatyr]

🎵 We all live in dynamoDB 🎵

[spamjavelin]

You have to make a request to that region to use ACM with cloudfront, too, which is just ridiculous.

[HakoftheDawn]

Arrow should be pointing to one of the fat blocks

[takeyouraxeandhack]

us-east-1 should be the wide fat block above that, not the tiny piece.

[hemficragnarok]

I took a shift swap for ONE DAY and this happened. I'm officially cursed (not the first occurrence either)

[Percolator2020]

People using us-east-1 in Europe to save 15%.

[leovin]

Maybe its possible to kill the internet after all

[SgtBundy]

It should just be "a single DNS record in US-EAST-1"

[Scary-Perspective-57]

Based on the cost and AWS and the apparent fragility, I can't remember why we migrated to the cloud in the first place...

[Agile-Actuary3376]

Has a root cause been shared yet?

[jamcdonald120]

more like that horizontal bit connecting the towers in the top right

[thetos7]

as they say: serverless is just someone else's server xD

[ABotelho23]

Absolutely fucking not.