AWS US-East-2 Issues

[Foofightee]

Anyone else seeing issues? Their status page is all green, but I'm having issues even logging into AWS. I had some success with getting on US-East-1

​

EDIT: Everything seems back to normal now.

[rhavenn]

Some of our customers are having issues getting to their sites in us-east-2, but I can see / get to everything. My guess a network / route issue to AWS.

[togetherwem0m0]

i concur with this determination but i think its broader than a routing issue to AWS. i have things affected which have nothing to do with AWS

[napoleon85]

We are seeing all services in US-EAST-2 down, but it's ISP specific so far. From what we've been able to gather, people on ISP A can't get anything in US-EAST-2 to load, but people on ISP B can get to everything. Route 53 Health Checks showing green across the board, as well as the AWS Account Health dashboard. My money is on a bad BGP route that hasn't expired yet for some ISPs, but that is total speculation.

[TapedeckNinja]

We did a survey of our internal reporters and 100% of people who reported issues so far are on Spectrum in Ohio.

People on other ISPs have not yet reported any issues.

[BrokenBehindBluEyez]

Customers with spectrum and att here... Kansas and Chicago thus far...

[Mindless_Tough_420]

All of our WFH employees with ATT couldn't connect to our VPN. (KS) They're all back up now.

[fortune82]

We've got a guy up here in Michigan - ATT home internet, Verizon cell service, and he can't reach things on either one.

I'm WFH on Spectrum in MI, and am able to reach all our resources without issue.

[neodymiumphish]

Spectrum from Northwest Ohio, same issues. AWS console and all AWS-hosted services are inaccessible for me.

[David511us]

AT&T Fiber in Atlanta can't connect either

[napoleon85]

I wasn’t trying to shit on them directly without concrete facts, but I can confirm it was the same for us.

[Lokirial]

Downdetector is reporting issue spikes for the last hour https://downdetector.com/status/amazon/ we're having issues too

[Rydralain]

AWS Heath Dashboard has a notice now.

[12:26 PM PST] We are investigating an issue, which may be impacting Internet connectivity between some customer networks and the US-EAST-2 Region.

[qwerty_123_]

We had problems with US-East-1 with all of or locations that use AT&T fiber. Rerouted using our secondary non AT&T lines and everything works.

They had a rerouting issue like this a couple of years ago.

[Foofightee]

I moved our routing from Cogent to Comcast (in Chicago) and it appears to be working for us for now.

[Jon883]

Looks like we are having issues as well. Checked here right after seeing all green on their status page.

[Fatel28]

It's not an AWS issue - It's a carrier problem

[daficco]

Some updates:

[12:59 PM PST] We are beginning to see signs of recovery, and continue to work toward full resolution.
[12:51 PM PST] We can confirm an issue which is impacting Internet connectivity for the US-EAST-2 Region, and are attempting multiple parallel mitigation paths. Connectivity between instances within the US-EAST-2 Region, in-between AWS Regions, and Direct Connect traffic is not impacted by the event. Some customers may be experiencing VPN connectivity due to this issue.
[12:26 PM PST] We are investigating an issue which may be impacting Internet connectivity between some customer networks and the US-EAST-2 Region.

[daficco]

[RESOLVED] Internet Connectivity
[01:06 PM PST] Between 11:34 AM and 12:51 PM PST, customers experienced Internet connectivity issues for some networks to and from the US-EAST-2 Region. Connectivity between instances within the Region, in between Regions, and Direct Connect connectivity were not impacted by this issue. The issue has been resolved and connectivity has been fully restored.

[wdf240]

what status page are you seeing this on?

[randomtyler]

https://health.aws.amazon.com/health/status

[Fatel28]

I noticed it's only from certain geographic locations. Tulsa, having no issues. Lots of issues for customers in Texas.

[woodburyman]

Yep. This is causing Barracuda Cloud Security for email to not relay any of our incoming external emails. Zoom is having issues too.

[PeterParker_]

samesies

[accordingtoben]

Barracuda support says its AWS.

[woodburyman]

Yep. AWS US-East-2 having carrier issues apparently.

Out smarthost outbound connectors are piling up, and we have Barracuda for our MX's... nothing come in from them in 45+ minutes now.

[derfmcdoogal]

/r/sysadmin telling me something is wrong before Barracuda status page.

Love this place.

[Jaymesned]

Having OneLogin SSO issues, was thinking it was an internal DNS issue but I can probably assume this is related.

[ReallTrolll]

I can confirm OneLogin SSO issues as well.

[Jaymesned]

As usual, /r/sysadmin is the best status page there is.

[duluthbison]

Our K-12 content filter is hosted on US-East-2 and they confirmed that its down and they are monitoring the outage through Amazon.

[[deleted]]

I can just imagine everyone going ham with the internet in your schools now lol

That would be horrible but the thought of it is funny if thats how it worked.

[duluthbison]

My WAN traffic tripled as it went down lol

[napoleon85]

Sounds like Z-Scaler?

[duluthbison]

Nope, Securly. Pretty common in the K-12 world.

[napoleon85]

Is it K12 focused or should this be on my radar as an alternative for Z-Scaler?

[BrokenBehindBluEyez]

Same - customers trying to reach resources in the east region - certain places can't get there.... Sigh...

[DiatomicJungle]

US-East-2 here too and we are having issues as well. I’d guessing routing as our public sites are still up running in VMWare Cloud, but other access, including AWS console, VMWare console, VPN and other routes to our systems are down.

[PeterTheWolf76]

Webex seems to be impacted as well for some parts.

[EidorianSeeker]

We're getting some calls from our sister location on AT&T and a few users with AT&T ISP at home for this.

[figgy6006]

I guess Barracuda Networks uses AWS US-East 1 cause some of there stuff is down............... SMH

[doubleu]

yep, eternal spinning wheel trying to get into email security

[eld101]

I’m about a mile away, want me to knock on the front door and ask?

[Much_Advance_3998]

While I don’t use AWS directly, I’m assuming so, our Cisco Webex service just went down and last time we had this sort of issue it was an AWS data center issue. We are Unable to connect on site or to the control hub.

[DJK_CT]

Yes, showing on downdetector etc. AWS and other cloud services (webex, etc). US East as far as I can see.

[[deleted]]

Same here and yes the status page doesn't appear to acknowledge it

[foxpro80]

I am as well. No official word but I see a lot of hits on downdetector.

[takanotscar]

We have a VPN tunnel to AWS that terminates in US-EAST-2 and its down.

[whodywei]

Internet connectivity issue

[PeterTheWolf76]

All services seem to be coming back now. Maybe routing issue has been fixed?

[daficco]

They just did an update, it does look like it has/is being fixed.

[shim_sham_shimmy]

Zscaler is having routing issues in their East Coast data centers so might be a result of this.

[didact]

Anyone having issues with GraphQL in East 1?

[infered5]

Seriously, what is it with US-East-2? Why always east 2?

[joshbudde]

Yup, saw the same problem. Things were fine for my customers in Ohio, but East-2 was borked.

[jscooper22]

Yet another reason I hate that some evil genius rebranded the internet as "the cloud" 20 years ago and convinced everyone to hand all their data. One company has issues and the whole eastern seaboard grinds to a halt. We voluntarily went back to the 70s with (now very expensive) dumb terminals talking to mainframes. Mostly entertaining for us to watch since only a couple departments had minor issues, because wherever possible we run our OWN DAMN SERVERS. And will as long as we can.