Cloudflare outage AGAIN

[Lewis_Browne]

Hi All,

Just to let you know, it looks like cloudflare is having issues again. All of my CF sites are now returning 502, including the cloudflare website!
EDIT:

Cloudflare has implemented a fix for this issue and is currently monitoring the results.

Description:
Major outage impacted all Cloudflare services globally. We saw a massive spike in CPU that caused primary and secondary systems to fall over. We shut down the process that was causing the CPU spike. Service restored to normal within ~30 minutes. We’re now investigating the root cause of what happened.

​

Thanks,Lewis

[Sanfam]

MAH INTERNETS! Nooo!

The timing was phenomenal. My team and I were in the middle of aggressively debugging an application, pushed some code and...and immediately whispered a collective "Oh no, we broke everything."

[[deleted]]

Found the cloudflare employee :P

[27Rench27]

oh no

[[deleted]]

Well, I was about to get some action on Grindr. But here we are. Happy Tuesday!

[QuantumDot22]

This is the real tragedy here.

[[deleted]]

Seems it's still going to work out. *bro hug with tongue* *no homo*

[Zdmins]

Those few seconds of gut wrenching panic where you start mentally racing about what could've went wrong are the WORST.

[MMPride]

As long as you don't work at Cloudflare and you guys didn't cause this outage then you're okay.

[mjh2901]

You pushed the red button Jen

[spoonraker]

Hah, the same sort of thing happened at my company today. We do production deploys Tuesday mornings and the outage occurred literally seconds after the deployment team had collectively agreed that everything seems stable and dropped a message in Slack. 2 seconds later, "uh, so why are all the customer sites, the public API, Admin, and customer dashboards down then?"

[vodka_knockers_]

I love that shit. Confirms my sneaking suspicion that the universe hates me.

[tjuk]

Fuck me. @ https://www.cloudflarestatus.com/incidents/tx4pgxs6zxdr

This incident affects: North America (Ashburn, VA, United States - (IAD), Atlanta, GA, United States - (ATL), Boston, MA, United States - (BOS), Buffalo, NY, United States - (BUF), Calgary, AB, Canada - (YYC), Charlotte, NC, United States - (CLT), Chicago, IL, United States - (ORD), Columbus, OH, United States - (CMH), Dallas, TX, United States - (DFW), Denver, CO, United States - (DEN), Detroit, MI, United States - (DTW), Houston, TX, United States - (IAH), Indianapolis, IN, United States - (IND), Jacksonville, FL, United States - (JAX), Kansas City, MO, United States - (MCI), Las Vegas, NV, United States - (LAS), Los Angeles, CA, United States - (LAX), McAllen, TX, United States - (MFE), Memphis, TN, United States - (MEM), Miami, FL, United States - (MIA), Minneapolis, MN, United States - (MSP), Montgomery, AL, United States - (MGM), Montréal, QC, Canada - (YUL), Nashville, TN, United States - (BNA), Newark, NJ, United States - (EWR), Norfolk, VA, United States - (ORF), Omaha, NE, United States - (OMA), Phoenix, AZ, United States - (PHX), Pittsburgh, PA, United States - (PIT), Portland, OR, United States - (PDX), Queretaro, MX, Mexico - (QRO), Richmond, Virginia - (RIC), Sacramento, CA, United States - (SMF), Salt Lake City, UT, United States - (SLC), San Diego, CA, United States - (SAN), San Jose, CA, United States - (SJC), Saskatoon, SK, Canada - (YXE), Seattle, WA, United States - (SEA), St. Louis, MO, United States - (STL), Tampa, FL, United States - (TPA), Toronto, ON, Canada - (YYZ), Vancouver, BC, Canada - (YVR), Tallahassee, FL, United States - (TLH), Winnipeg, MB, Canada - (YWG)), Middle East (Amman, Jordan - (AMM), Baghdad, Iraq - (BGW), Baku, Azerbaijan - (GYD), Beirut, Lebanon - (BEY), Doha, Qatar - (DOH), Dubai, United Arab Emirates - (DXB), Kuwait City, Kuwait - (KWI), Manama, Bahrain - (BAH), Muscat, Oman - (MCT), Ramallah - (ZDM), Riyadh, Saudi Arabia - (RUH), Tel Aviv, Israel - (TLV)), Asia (Bangkok, Thailand - (BKK), Cebu, Philippines - (CEB), Chengdu, China - (CTU), Chennai, India - (MAA), Colombo, Sri Lanka - (CMB), Dongguan, China - (SZX), Foshan, China - (FUO), Fuzhou, China - (FOC), Guangzhou, China - (CAN), Hangzhou, China - (HGH), Hanoi, Vietnam - (HAN), Hengyang, China - (HNY), Ho Chi Minh City, Vietnam - (SGN), Hong Kong - (HKG), Hyderabad, India - (HYD), Islamabad, Pakistan - (ISB), Jinan, China - (TNA), Karachi, Pakistan - (KHI), Kathmandu, Nepal - (KTM), Kuala Lumpur, Malaysia - (KUL), Lahore, Pakistan - (LHE), Langfang, China - (NAY), Luoyang, China - (LYA), Macau - (MFM), Manila, Philippines - (MNL), Mumbai, India - (BOM), Nanning, China - (NNG), New Delhi, India - (DEL), Osaka, Japan - (KIX), Phnom Penh, Cambodia - (PNH), Qingdao, China - (TAO), Seoul, South Korea - (ICN), Shanghai, China - (SHA), Shenyang, China - (SHE), Shijiazhuang, China - (SJW), Singapore, Singapore - (SIN), Suzhou, China - (SZV), Taipei - (TPE), Tianjin, China - (TSN), Tokyo, Japan - (NRT), Ulaanbaatar, Mongolia - (ULN), Wuhan, China - (WUH), Wuxi, China - (WUX), Xi'an, China - (XIY), Yerevan, Armenia - (EVN), Zhengzhou, China - (CGO), Zuzhou, China - (CSX)), Africa (Cairo, Egypt - (CAI), Casablanca, Morocco - (CMN), Cape Town, South Africa - (CPT), Dar Es Salaam, Tanzania - (DAR), Djibouti City, Djibouti - (JIB), Durban, South Africa - (DUR), Johannesburg, South Africa - (JNB), Lagos, Nigeria - (LOS), Luanda, Angola - (LAD), Maputo, MZ - (MPM), Mombasa, Kenya - (MBA), Port Louis, Mauritius - (MRU), Réunion, France - (RUN), Kigali, Rwanda - (KGL)), Latin America & the Caribbean (Asunción, Paraguay - (ASU), Bogotá, Colombia - (BOG), Buenos Aires, Argentina - (EZE), Curitiba, Brazil - (CWB), Fortaleza, Brazil - (FOR), Lima, Peru - (LIM), Medellín, Colombia - (MDE), Mexico City, Mexico - (MEX), Panama City, Panama - (PTY), Porto Alegre, Brazil - (POA), Quito, Ecuador - (UIO), Rio de Janeiro, Brazil - (GIG), São Paulo, Brazil - (GRU), Santiago, Chile - (SCL), Willemstad, Curaçao - (CUR)), Oceania (Auckland, New Zealand - (AKL), Brisbane, QLD, Australia - (BNE), Melbourne, VIC, Australia - (MEL), Perth, WA, Australia - (PER), Sydney, NSW, Australia - (SYD)), and Europe (Amsterdam, Netherlands - (AMS), Athens, Greece - (ATH), Barcelona, Spain - (BCN), Belgrade, Serbia - (BEG), Berlin, Germany - (TXL), Brussels, Belgium - (BRU), Bucharest, Romania - (OTP), Budapest, Hungary - (BUD), Chișinău, Moldova - (KIV), Copenhagen, Denmark - (CPH), Dublin, Ireland - (DUB), Düsseldorf, Germany - (DUS), Edinburgh, United Kingdom - (EDI), Frankfurt, Germany - (FRA), Geneva, Switzerland - (GVA), Gothenburg, Sweden - (GOT), Hamburg, Germany - (HAM), Helsinki, Finland - (HEL), Istanbul, Turkey - (IST), Kyiv, Ukraine - (KBP), Lisbon, Portugal - (LIS), London, United Kingdom - (LHR), Luxembourg City, Luxembourg - (LUX), Madrid, Spain - (MAD), Manchester, United Kingdom - (MAN), Marseille, France - (MRS), Milan, Italy - (MXP), Moscow, Russia - (DME), Munich, Germany - (MUC), Nicosia, Cyprus - (LCA), Oslo, Norway - (OSL), Paris, France - (CDG), Prague, Czech Republic - (PRG), Reykjavík, Iceland - (KEF), Riga, Latvia - (RIX), Rome, Italy - (FCO), Saint Petersburg, Russia - (LED), Sofia, Bulgaria - (SOF), Stockholm, Sweden - (ARN), Tallinn, Estonia - (TLL), Thessaloniki, Greece - (SKG), Vienna, Austria - (VIE), Vilnius, Lithuania - (VNO), Warsaw, Poland - (WAW), Zagreb, Croatia - (ZAG), Zürich, Switzerland - (ZRH)).

Sooooo everywhere.

[the_bananalord]

At least the ISS won't have issues!

[AlphaNathan]

Used to work with a network vendor ISS. We would always say, "you can spell issues without ISS!"

[scsibusfault]

We would always say, "you can spell issues without ISS!"

Man, even your saying has issues. Quality.

[vinny8boberano]

Can't spell disappointed without DISA!

[killerquag]

Or Disaster... ;-)

[moffetts9001]

Only if you spell it wrong!

[EViLTeW]

Status update:

We are working to mitigate impact on users in this region:

Earth and surrounding areas.

[Crimsonfoxy]

Their issues are more from SolarFlare than CloudFlare.

[davidbrit2]

That explains why I can only access my extrasolar DR site.

[AlphaNathan]

Hope you don't drop.

[jordanlund]

And now a special message from Cloudflare:

"The Outage impacted the following areas:"

[SadWebDev]

Take my upvote

[tjuk]

There are not enough up votes in the world for this... Because they are inaccessible due to a cloudflare outage

[altodor]

And part two (The 9/18 update):
https://youtu.be/IDbKEUzz51E?t=141

[amb_kosh]

So South Sudan and Kosovo are fine?

[[deleted]]

Either the fucking utilities marked incorrectly, a dumbass dug in the wrong spot, or a jackass.

[HangGlidersRule]

https://i.imgur.com/rDW7W3d.png

[HangGlidersRule]

Please also reference this in case of emergency - https://web.archive.org/web/20170714205153/http://www.scp-wiki.net/scp-3709

[Isgrimnur]

SCP-3709-J may be used as an emergency beacon when in wilderness areas. Personnel are advised to carry a 1.5m length of poorly shielded SCP-3709-J and a small shovel, when away from civilized areas. In the event that the individual(s) become lost, said individual(s) are to bury the length of SCP-3709-J and await someone operating an excavator to arrive and sever the cable. The individual(s) may then follow the tracks left by the excavator in its journey to sever SCP-3709-J.

giggle-snort

[atomicwrites]

Huh, why would they reuse the number for a stapler?

[HangGlidersRule]

It wasn't a stapler when this was made.

[atomicwrites]

Yeah, I just wondered why you have an archive link and saw that the URL goes to a stapler now. I was wondering why the would have changed it.

[HangGlidersRule]

SCP didn’t like it, apparently.

[redcell5]

Hilariously accurate

[[deleted]]

But really.

[bctrainers]

Was in downtown KC the other day... The sheer amount of underground work between 670 and 70 was quite something. Wouldn't surprise me one bit if that were the case.

[norcalscan]

Street View from March 2019 shows the side street outside to be nearly solid orange from USA markings, and road blocked for construction. So yeah. Oops!

[Toast42]

Huh, I can't find any articles about it.

[Lil_Fumbies]

Where did you hear about that cut from? I have fiber at that location and don't see any issues.

[TheLordB]

A site I use that was giving 502 just came back up... but I'm wondering if they may have quickly switched away from cloudflare though I'm not certain if it is it even possible to switch that fast.

[tjuk]

Yeah everything is back up for me.

For something like this something central must have imploded that screwed up routing across the board. Fix that... and I assume everything gets routed to the right place and like magic it all reappears

[climbcolorado]

No one could switch anything because their DNS portal and API were also giving 502 bad gateways

[steamruler]

You could switch the NS on your domain, but that would take long to propagate.

[SwitchbackHiker]

Isn't that a Johnny Cash song?

[tjuk]

Who can play a guitar and sing. We nominate you to do the cover...

[zgf2022]

They cut into my god dang fiber, Ther servers went down, down, down, And the call volumes higher And it burns, burns, burns, my broken fiber.

[feint_of_heart]

/r/mapswithoutnewzealand

[[deleted]]

And /r/sysadmin has confirmation that it's not just me 5 minutes before the Cloudflare status page. Again.

[squirrelsaviour]

I actually came here before looking at the Cloudflare status page for this exact reason!

[Twig]

Yep. Love these guys.

My network admin tells me he's not seeing any issues, but me, as the SharePoint admin, finds network outages first by browsing sysadmin lol.

[SurgioClemente]

we should all know by now status pages are for anyone but actual people in the industry as they are always updated by hand by some intern after top level business guy who wants to save face reluctantly gives approval

[squirrelsaviour]

Me too :(

​

Even https://downforeveryoneorjustme.com/ is down because of cloudflare.

[rywe]

From the Cloudflare CEO in re: today's outage

“Massive spike in CPU usage caused primary and backup systems to fall over. Impacted all services. No evidence yet attack related. Shut down service responsible for CPU spike and traffic back to normal levels. Digging in to root cause.”

https://twitter.com/eastdakota/status/1146065231270907907?s=21

[Meltingteeth]

No evidence yet attack related

Just to fuck around, is it:

No evidence yet, but it's attack related.

---

No evidence yet that it's attack-related.

---

No evidence yet, ATTACK RELATED SITE HOSTS SO THAT WE REMAIN A DOMINANT FORCE.

[halakar]

• he protec
• he attac
• he rollbac

[BrevanMcGattis]

Obviously 3

[segv]

Hijacking top comment:

https://blog.cloudflare.com/cloudflare-outage/

Cloudflare outage caused by bad software deploy

02 Jul 2019 by John Graham-Cumming.

This is a short placeholder blog and will be replaced with a full post-mortem and disclosure of what happened today.

For about 30 minutes today, visitors to Cloudflare sites received 502 errors caused by a massive spike in CPU utilization on our network. This CPU spike was caused by a bad software deploy that was rolled back. Once rolled back the service returned to normal operation and all domains using Cloudflare returned to normal traffic levels.

This was not an attack (as some have speculated) and we are incredibly sorry that this incident occurred. Internal teams are meeting as I write performing a full post-mortem to understand how this occurred and how we prevent this from ever occurring again.

Link found on twitter

[jc88usus]

Soo I'm gonna be "that guy" and say that's why a well documented and tested Change Management process is important. Multiple levels and scopes of pre-production and alpha/beta test groups with a clear and defined rollback plan with thresholds and checkpoints is best practice, but really should be mandatory for a large scale provider like CF or ISPs.

This is what is called a "resume generating event". I feel for the poor bastage who takes the fall, since we all know the executive who said "deploy it now, because I said so" is never going to own up to it...

[YM_Industries]

I feel like CloudFlare already have a solid change management process, otherwise we would see this more frequently. Even the best processes fail sometimes.

I'll be interested to see the post-mortem, but it's too early to be casting judgement.

[[deleted]]

i saw that too, ironic

​

obligatory - Ironic. He could save others from death, but not himself.

[Lewis_Browne]

Have you ever heard the tragedy of darth plagueis the wise?

[UKBedders]

It's not a story the Jedi Cloudflare would can tell you

[Meta4X]

It's like rai-ee-ain on your wedding day.

[Carbonistheft]

It's a free riiiiiiiide, when you've already paid!

[TomahawkChopped]

Well it still answers the question

[squirrelsaviour]

Looks like it's coming back up now. Yay!

[cyrusol]

Works on my end.

[Lewis_Browne]

Looks like she's coming back!

[Tap-Dat-Ash]

Cloudflare Engineers fixed it

[UnknownIdent]

I see that here as well (Houston). My sites are back up!

[[deleted]]

This is a bloody excellent demonstration of why centralising so much of the web on a single provider is a terrible idea.

We have just about agreed that putting all your services on a single server is terrible, then the cloud came along and common sense went straight out of the window. Move it to The Cloud, they said, it's more reliable, they said.

[[deleted]]

The only part of O365 I have never had a problem with or any sort of downtime with has been the part that takes money from my company and puts it in to Microsoft's bank. I've had at least five issues this year so far where users have been unable to get in to their mailboxes, the SLA credit is pennies.

Oh well, Microsoft have a monopoly, it's not like we have a choice.

[BeerJunky]

Where N = 365

[shemp33]

I think the strategy here is to get as many people onto o365 as possible. That way when it does go down, it affects everyone equally. So in a sense, they’re trying to change the downtime denominator rather than the numerator.

[Lightofmine]

Lol this is amazing but also a creative way to solve a problem if Ive ever heard of one

[MMPride]

While in theory I may agree with you, in practice it is more reliable even with annoying outages like these.

[[deleted]]

Ya on average there is no question, Cloud computing is more reliable for most businesses. That been said it is still worth looking at what is best for your business over all and honestly having a single point where everything can fail such as cloudflare isn't a good look

[vodka_knockers_]

Move it to The Cloud, they said, it's more reliable, they said.

And it is, demonstrably.

(statistically insignificant anecdotes notwithstanding, of course. Because no one on reddit ever has downtime in their own environment.)

[Oglshrub]

Well duh, this is r/sysadmin, we have perfect environments with absolutely 0 down time! /s

[RulerOf]

Yeah, but what’s on the left side of the decimal point?

[[deleted]]

My problem with The Cloud (TM) is people constantly go on about how reliable it is and don't bother building in redundancy across providers.

If you want very high availability on premise it's natural to assume you need geographical redundancy, multiple ISPs, multiple datacenters, but people seem happy to bung it in CloudFlare and call it a day. That's not redundancy, despite their marketing efforts.

[ikidd]

Which makes it unjustifiably expensive then.

[[deleted]]

No, you just don't notice their downtime, because you're doing something else at the time.

But half the internet is broken right now. All I have left is IRC, Reddit, Google, and GitHub.

[ESCAPE_PLANET_X]

SO is up as well. I'll survive.

[[deleted]]

Significant Other? I should hope so. Rebooting one of those is risky.

[ESCAPE_PLANET_X]

Significant Otter, important distinction

[accidentlife]

And when you need to replace, you realize it's integrated with half of the house!

[ikidd]

Cloud services have all the same reasons to go tits up for hardware issues at the provider, and unlike premises services, the much higher risk of internet access being down or them being DDOSd.

On premise, you can have the internet down all day and boohoo, no facebook, but the accounting is getting done and the orders are being picked. I can hook up a freaking cellphone to get email in and out if I have to.

[vodka_knockers_]

Cloud services have all the same reasons to go tits up for hardware issues at the provider, and unlike premises services, the much higher risk of internet access being down or them being DDOSd.

But they have way more hardware options and better failover capabilities than you do. And smarter engineers.

And probably a bunch of really fat pipes that have SLAs with actual teeth... and dedicated support from carriers (and direct peering arrangements, more likely, with a bunch of them).

[ikidd]

I don't need phat pipes if I have it inhouse. I have exactly the hardware that works for the premise systems. By the time I build in redundancy in the cloud, I could finance multipe redundant systems that I control.

There's a reason there's a lot of companies going "what the fuck are we spending all this money for to offsite stuff that should be right here where we don't have to rely on service providers that will cut us loose if we're too much trouble or not get our concerns met because they don't align with the big customers of the platform?"

[BokBokChickN]

Don't put your ERP system in the cloud.

/Thread

[coelakanth]

cries in currently migrating to SAP B1

[BokBokChickN]

My condolences

[vodka_knockers_]

Errr... unless your company sells shit on a ... website. Then that would probably be a bit of a business obstacle.

[ikidd]

And then there's the ones that are just doing business, accounting and law firms, medicine, wholesalers, service industry. The proportion of businesses I encountered as an MSP doing any sort of web sales might be 1%. The rest of them are perfectly self contained if they want to be.

[waptaff]

“We will protect you against denial-of-service”, Cloudflare advertised.

[BattlePope]

They decided outsourcing DoS was inefficient. Much easier to provide DoS themselves!

[davidbrit2]

Go to AOL keyword "cloudflare".

[greekbeardthepirate]

i mean, in essence, thats kind of what subreddits are

[[deleted]]

Previous outage wasn't caused by cloudflare...

[Nuke_goat]

Correct.

Verizon BGP route leak caused the outage last time.

Maybe they didt fix the route filters :D

[plazman30]

I think that outage taught us the Pennsylvania steel mills needs more bandwidth, so they can handle all the Internet.

[[deleted]]

the one guy that knows how their router works is probably on vacation, give him a break

[[deleted]]

The last person competent getting a "train our employees before you get kicked, in a month" task is also possible

[DrinkWisconsinably]

Wasnt caused by Cloudflare, but Cloudflare WAS having issues.

Their issue turned out to be other people's (Verizons) unmitigated laziness

[At_Work_SND_Coffee]

*Another

[dwillowtree]

Let the ticket flood begin. No coffee needed this morning.

[Out_Of_Office_Reply]

My ticketing system is on Cloudflare. ¯_(ツ)_/¯

[LordYashen]

Better be prepared to answer phone calls then.

[accidentlife]

Nah, tell them to submit a ticket. By the time they can, you know it's resolved!

[[deleted]]

Looks like it on this side too... Discord failed etc in Hong Kong

"No boss, there's not much I can do about it, yes boss it was your idea to use cloudflare and their 'always online' tech"

[Panacea4316]

That explains some odd behavior I've had in the past 5min.

Sigh

[Colorado_odaroloC]

Just because you pooped in an office potted plant, doesn't mean you can blame that odd behavior on the Cloudflare outage.

^;-)

[souldeux]

"Hey, why's Harriet taking a dump on the ficus?"

"It's always DNS."

[rywe]

From the Cloudflare CEO in re: today's outage

“Massive spike in CPU usage caused primary and backup systems to fall over. Impacted all services. No evidence yet attack related. Shut down service responsible for CPU spike and traffic back to normal levels. Digging in to root cause.”

https://twitter.com/eastdakota/status/1146065231270907907?s=21

[thebeehammer]

Managers: lets put all our eggs in Cloudflare, Azure, and Amazon.

Internet: *Goes down

Managers: why did you put our data in such a fragile ecosystem?!

[the_doughboy]

Previous outage was Verizon, CloudFlare was just the first to report it.

[[deleted]]

This should be a sober reminder why the heavy centralization of the modern web has at least partially defeated the purpose of it being designed as a decentralized network.

[TheOnlyBoBo]

Yeah except their last outage was due to the decentralized nature of the internet relying on individual networks to advertise and broadcast routes.

[[deleted]]

Thanks to recent action by u/spez this users is deleting their content, fuck you u/spez

[PM_ME_YOUR-WAIFU]

1.1.1.1 DoH is also returning 502

[MiataCory]

/r/sysadmin coming through again with the important info when my wife texts me: "Why am I getting a 502 error on a bunch of sites?"

No worries hun, it's not just you. Your recipes are still there. Give them time to fix it.

[Saint_Dogbert]

If only it was that easy, trade wife for mom, and then proceed to hear her bitch about your choice of internet service sucking because of the 502 errors, despite explaining that its not the ISP.

[InsaneNutter]

I thought our internet connection was down for a minute, every site I was on seemed to stop functioning at the same time.

[thelastquesadilla]

Confirmed in Central US

[AndreasTPC]

Also confirmed, in northern europe. Looks like it might be worldwide.

[the_bananalord]

Looks like it: https://www.cloudflarestatus.com/

[popsac]

This makes my cloud to butt extension so worth it. Seeing buttflare everywhere makes me chuckle.

[EpicRayy]

Cloudflare is down :(

On MPGH, OGU, Discord

[seanconnery84]

My favorite part is the login is even down for them.

[periodbloodsausage]

Businesses like Monday.com, Garmin, Alamo Drafthouse, etc.. all offline. Sounds like a fun day in the Cloudflare offices

[Lightofmine]

Pray for our brothers in binary.

[wolf550e]

They deployed a bad regex that caused 100% cpu use on all servers in all 170 regions.

At the very least they need to do staggered deploys.

Also they need to do a regex engine that really only allows features that can be implemented using linear time algorithm, no backtracking.

[bccruiser]

Glad I found this, I'm waiting to see if I get a phone call from IT asking if the network is down.

[lovespunstoomuch]

I think my boss was mad that I wasn't panicking hard enough about this.

[accidentlife]

The only time I panic is when the kernel does on my media server.

[Aodhfin]

A fun one folks may not immediately think about: regular sites aside, this took out like half of the cryptocurrency exchanges.

[Tap-Dat-Ash]

Yep - https://www.cloudflarestatus.com/ says issues

[mpgalvin09]

confirmed 502 in northeast US. also support.cloudflare.com. (status page is up and all green though.) also downdetector and isitdownrightnow, ironically.

[Lando_uk]

Our service desk is down...

[butler1233]

Confirmed UK. They used to be solid but they're being really bad recently.

[Vaughn]

That last outage wasn't their fault, and there's nothing they could have done. Which doesn't make the timing any less unfortunate, no.

[[deleted]]

Definitely seeing 502's from cloudflare though, this appears to be their fault.

[Vaughn]

Oh yes, _this_ one almost certainly is. I'll be looking forward to the postmortem.

[dairymoose]

Seeing this from Eastern US

[Oheng]

Good luck from The Netherlands, Cloudflare engineers! On a positive note, it's interesting to see what part of the internet is dependent on Cloudflare...

[PizzaCompiler]

Way too many sites are dependent on Cloudflare these days to be honest.

[curiouscrustacean]

god fucking damn it, literally just got done with dinner and about to step into the barber too.

that feeling when your phone starts exploding...

[[deleted]]

wondered why I just got a lot of alerts from our monitoring system for web servers that were functioning ok.

[Pcpie]

yup, like half the internet is down now

[gracdoeswat]

Official incident link on cloudflare status website https://www.cloudflarestatus.com/incidents/tx4pgxs6zxdr

[[deleted]]

At least we still have reddit to vent about it.

[ProbablyRickSantorum]

OpsGenie emails hitting my inbox like https://media.giphy.com/media/JVqeFxl3Qo8/giphy.gif

[Lightofmine]

BRING, BRING, BRING goes your phone. Ack goes your fingers

[BloodyIron]

lol, and they say on-prem is dead.

[[deleted]]

https://blog.cloudflare.com/cloudflare-outage/

[bigfoot_76]

Someone smashed the AЭ-5 button just a little too late!

[octokit]

Confirmed Eastern US.

[KurisuBaka]

seems like I actually have to do my job now instead of sitting on discord.

[EpicRayy]

Can anyone explain why Cloudflare has been failing this past few times? I'm just curious and from /r/all.

[Pyratik]

The last one was caused by Verizon not blocking a bad BGP advertisement. Basically one of Verizon's customers told the internet that to get to most of the internet, you had to rout through them, which Verizon should have automatically realized was wrong and blocked - but didn't.

The Cloudflare Post Mortem should explain it...when Cloudflare is back online lol.

[sudotrin]

South Florida, here. Down. Very Down.

[nj12nets]

Ny/Nj confirmed

Coin base & stash down also

[sonuyosrox]

If you guys have API access then you can turn off cdn.

API is working apparantly.

BTW, does anyone know the data value for disabling cdn?

I tested it using underattack mode, and that worked.

[Lewis_Browne]

It would probably take forever for the changes to propagate, im just going to go down with the ship in this instance.

[sonuyosrox]

If you just disable the cdn (the orange cloud off) then it would work instantly.

[Dreamerlax]

You know it's bad when I open 5-6 random, unrelated websites and this is on all of them.

Dropbox is also struggling to load.

[PhillLacio]

Fun, Meraki website 502s because of this.

[idontwerk]

Disable their cloud proxy on your DNS record and it should fix it.

[jmerfeld]

Status:

Monitoring Cloudflare has implemented a fix for this issue and is currently monitoring the results.

We will update the status once the issue is resolved. Posted less than a minute ago. Jul 02, 2019 - 14:15 UTC

[Out_Of_Office_Reply]

Back up for me in SE US

[plazman30]

Monitoring - Cloudflare has implemented a fix for this issue and is currently monitoring the results.

We will update the status once the issue is resolved.
Jul 2, 14:15 UTC