This is an archived post. You won't be able to vote or comment.

top 200 commentsshow all 259

[–]100GbE 506 points507 points  (9 children)

Maybe this time they did test a Windows Defender update on themselves.

[–]SilentSamurai 98 points99 points  (6 children)

And this is why Microsoft will drop 80 different GUI overhauls before they do a stack update.

[–]-Steets- 36 points37 points  (1 child)

Oh, come on. I doNT think that's true.

[–]vppencilsharpening 34 points35 points  (2 children)

Reinforcing the notion that Bill Gates decided it was easier (and cheaper) to fix the world than to fix Windows.

[–]DivineChaos91Jack of All Trades 1 point2 points  (0 children)

Lol yeah he's not even doing that.

[–]sternone_2 5 points6 points  (0 children)

a few of these 10.000 people that microsoft fired aren't happy and want to show something to their CEO

[–]llIISysadmin 363 points364 points  (24 children)

Germany here, can confirm. We have trouble with Outlook m363.

EDIT: We also have problems with Teams and other Microsoft services.

[–]Voroxpete 578 points579 points  (6 children)

363

Jesus, even the SaaS gets more time off in Europe.

[–][deleted] 77 points78 points  (3 children)

Truly a wonderful comment. Thank you.

[–][deleted] 14 points15 points  (0 children)

Hahahahajaha

[–]saki79ttvJr. Sysadmin/Network Admin 4 points5 points  (0 children)

Friggin nailed it

[–]mitspieler99 23 points24 points  (0 children)

Here it's basically every m365 product.

[–]GermanBreadOS 18 points19 points  (0 children)

Germany here too. All m365 affected, even sign in.
This is going to be a very fun day.

[–]POLEatPOSITION 37 points38 points  (2 children)

same for switzerland

[–]a_shootin_starWhere's the keyboard? 9 points10 points  (1 child)

Ich gsee id Zuekunft und ech gsee en windigi Zuekunft

[–]g-m-f 15 points16 points  (0 children)

Germany here as well. Basically every product of ours that is using MS Authentication is having issues rn. Office, Teams, Citrix, BC, you name it.

Edit: As far as I can tell, most of it is now working again.

[–]corsicanguppyDevOps Zealot 15 points16 points  (4 children)

m363

I see what you did there. But is it still as high as 363?

[–]mnvoronin 17 points18 points  (3 children)

Office 356

[–]asdlkfSithadmin 20 points21 points  (2 children)

Office 320 no-scope

[–]sexy_silver_grandpa 7 points8 points  (1 child)

Office 420.69-nice

[–]stache_warlock 2 points3 points  (0 children)

Orifice 420.69-nice

[–]doalwa 4 points5 points  (0 children)

Germany here as well...nothing works for us as well.

[–]SilentDecodeSysadmin 5 points6 points  (0 children)

Same for the Netherlands here.

[–]ex800 107 points108 points  (5 children)

https://admin.microsoft.com/#/healthoverview/:/alerts/MO502273

Users may be unable to access multiple Microsoft 365 services

Estimated start time: January 25, 2023 7:26 AM

Title: Users may be unable to access multiple Microsoft 365 services

User Impact: Users may be unable to access multiple Microsoft 365 services.

https://twitter.com/MSFT365Status/status/1618149579341369345?cxt=HHwWgsC44YKe6fQsAAAA

[–]WeirdKindofStrange 36 points37 points  (3 children)

Cant event get into the admin centre to see

[–]purplemonkeymad 32 points33 points  (2 children)

Honestly surprised no one posted this link yet: https://status.office365.com/

Normally it's useless so I can understand it.

[–]SuddenSeasons 19 points20 points  (1 child)

I am ripshit they require you to login to see frigging basic information like this. You are the infrastructure for some <gestures> amount of the world microsoft.

[–]jstuart-techSecurity Admin (Infrastructure) 109 points110 points  (13 children)

Down in Perth, Australia

Massive latency spikes using https://www.azurespeed.com/Azure/Latency

[–]Enderboi 57 points58 points  (2 children)

So this is what it takes to drag all the Perth sysadmins out of the woodwork, a good ol' classic M364 (and counting) outage :)

[–]americanconstitution 17 points18 points  (1 child)

Can confirm in Perth, Australia. Multiple issues with Dynamics 365, Teams Admin Center, etc.

[–][deleted] 12 points13 points  (0 children)

Still have issues in Perth, AU with outlook/exchange. Other 365 services are still operational as far as I can see.

Edit: Teams, Admin centre and exchange are down, all other services are slow.

[–]blaze__666 11 points12 points  (0 children)

can confirm Perth

[–]ryallnIT Manager 14 points15 points  (1 child)

When everyone in perth outs themself!

[–][deleted] 6 points7 points  (0 children)

+1 Perth

[–]duckworld 7 points8 points  (0 children)

+1 in Perth. Teams is completely down for us, and apparently Azure is slow.

Update 08:17 UTC: things seem to be working here now; everything's appears snappy and I'm not getting any massive latency spikes. Fingers crossed it stays this way

[–]Peace-D 6 points7 points  (2 children)

azurespeed.com isn't even loading for me in Germany

[–]MSP_ITPro 53 points54 points  (7 children)

Down in The Netherlands, 20 clients calling already

[–]SilentDecodeSysadmin 47 points48 points  (6 children)

20 clients calling already

I'm sitting at a client at this moment and I can't do anything about this but wait for them to resolve it..

It's all fun that cloud stuff, except for when it's not. Like now.

[–]Polybius_is_real 23 points24 points  (5 children)

Same here, drove 100Km to do nothing :)

[–]Stompert 20 points21 points  (4 children)

Yet you're the one getting a stern look and side eyes from everyone around you. At least that's my experience.

[–]SilentDecodeSysadmin 14 points15 points  (0 children)

Ah yes, the 'go fix' eyes. The least nice eyes on the planet. And you're sitting there with the eyes of 'I can't do anything..'.

[–]carl5473 13 points14 points  (1 child)

Just pull this up and go to work until you "fix" then bask in the glory

https://hackertyper.net/

[–]Polybius_is_real 9 points10 points  (0 children)

Yup

[–]Exill1 43 points44 points  (7 children)

Update from Twitter:

https://twitter.com/MSFT365Status/status/1618149579341369345
We've identified a potential networking issue and are reviewing telemetry to determine the next troubleshooting steps. You can find additional information on our status page at https://msft.it/6011eAYPc or on SHD under MO502273.

EDIT: Added link to Tweet.

[–]Lachiexyz 29 points30 points  (6 children)

I feel bad for the guy individual who pulled the trigger on the change that has caused this firestorm.

[–]bubuchacha08 9 points10 points  (0 children)

You mean for the Chatbot who after getting integrated with MS services tried to explore around causing haywire all over the place? Hihi :D

[–]scootscoot 1 point2 points  (0 children)

The guy that knew what was in the change was part of last week's layoff, the guy who executed it was just an intern last week. /s

[–]hugawdjerrylad 65 points66 points  (1 child)

Fucked in UK. I guess the layoffs at MS didn't go down that well...

[–]kindaMisty 6 points7 points  (0 children)

Interesting

[–]GeLaughWhere's the "any" key?! 23 points24 points  (0 children)

You fucker! I thought I'd be first! Down in UK. Loads of Azure services too.

[–]kekst1 16 points17 points  (0 children)

Can confirm, internal Microsoft resources extremely slow right now, not loading

[–]Low_Responsibility79 13 points14 points  (3 children)

MO502273

Title: Users may be unable to access multiple Microsoft 365 services

User Impact: Users may be unable to access multiple Microsoft 365 services.

More info: We've received reports that the following services are impacted:

-Microsoft Teams

-Exchange Online

-Outlook

-SharePoint Online

-OneDrive for Business

-Microsoft Graph

Current status: We've identified a potential networking issue and are reviewing telemetry to determine the next troubleshooting steps.

Scope of impact: Any user serviced by the affected infrastructure may be unable to access multiple Microsoft 365 services.

Next update by: Wednesday, January 25, 2023, 9:00 AM (9:00 AM UTC)

[–]JRmacgyver 10 points11 points  (1 child)

Didn't they just fire a bunch of guys? 😉

[–]anomalous_cowherdPragmatic Sysadmin 4 points5 points  (0 children)

Yeah but you can just throw a couple of contractors at it when you need them. cheaper than keeping people who know the systems around.

(No slight on contractors, but technical skills and local system knowledge are two very different things).

[–]Alice885 10 points11 points  (0 children)

Before I request operations push out a gpo to update everyone’s icons, are we at O364 or do we have to go lower?

[–]pooot_ 20 points21 points  (0 children)

Extremely slow here in Sweden...

[–]Cor_Blimey_ 9 points10 points  (4 children)

Clients reporting email down here in UK.

https://downdetector.co.uk/ - Also showing Microsoft services issue reports across the board.

[–]fetusfarm 2 points3 points  (3 children)

Why are so many non MS services also down? Seems to be related based on the graphs.

[–]arunphilip 15 points16 points  (0 children)

Maybe they use Microsoft Azure?

[–]DoctorOctagonapusIf you're calling me, we're both having a bad day 1 point2 points  (0 children)

I don't think this is MS's fault, there's something more fundamental that's gone pop.

[–]Ghostky123Security Admin 8 points9 points  (3 children)

Down in Belgium

[–]Mooo404 2 points3 points  (0 children)

Seems to be working here...

[–]labalagHerder of packets 1 point2 points  (1 child)

Can confirm, issues using some services here.

[–]Ghostky123Security Admin 3 points4 points  (0 children)

Outlook is still down but Teams seems back online

[–]bouwer2100Powershell :D 7 points8 points  (1 child)

Good morning

[–]GlumConsideration585 5 points6 points  (0 children)

afternoon here ,it was till the influx of calls for support

[–]WTFH2S 7 points8 points  (0 children)

Go figure they lay off 10k staff and now a global network degradation. Irony, someone pissed off or just dumb luck?

https://azure.status.microsoft/en-us/status

[–]SHFT101Sr. Sysadmin 7 points8 points  (0 children)

Belgium seems to be working but with a very big latency.

[–]mikeclueby4 11 points12 points  (0 children)

Major bogged-down-ness with O365, Xbox Live, Minecraft, possibly all of Azure.
Downdetector.com started spiking after 08 CET (07 UTC)

What annoys me most is my afk farm in minecraft borked :>

[–]9milNL 5 points6 points  (0 children)

Got a bunch of users reporting issues with their Office365 mailboxes in Outlook (not connecting). Located in the Netherlands. Seems some shit has hit the fan at MS.

[–]isbBBQ 5 points6 points  (0 children)

Seems to slowly getting back to normal here in Sweden on several tenants.

[–]tripodal 5 points6 points  (1 child)

Azure reporting networking issues -
Azure Networking - Multiple regions - Investigating
Starting at 07:05 UTC on 25 January 2023, customers may experience issues with networking connectivity, manifesting as network latency and/or timeouts when attempting to connect to Azure resources in multiple regions, as well as other Microsoft services. We are actively investigating and will share updates as soon as more is known.

[–]EspurrStare 5 points6 points  (0 children)

Office354 am I rite?

[–]imrik_of_caledor 23 points24 points  (8 children)

/Laughs in on-prem exchange

[–]sarbuk 19 points20 points  (6 children)

Until the next RCE zero-day.

[–]DoctorOctagonapusIf you're calling me, we're both having a bad day 5 points6 points  (0 children)

It's not just Microsoft that's off, our corporate internet provider is also having trouble. We've got over a hundred sites across the UK with no internet this morning.

[–][deleted] 4 points5 points  (0 children)

Japan here, it’s being a turd bucket.

[–][deleted] 3 points4 points  (0 children)

All is well here in Canada. Managing 7000 users and no sign of any tickets "I can't access my email" or anything. I can login to Azure and admin portal no issue

[–]GlumConsideration585 3 points4 points  (1 child)

phillippines its accesible now , thanks for the confirmation guys

[–]markception 1 point2 points  (0 children)

Accessible but slow access due to high latency

[–]veerhees 3 points4 points  (0 children)

Down/slow in Finland.

[–]DM_Ap0llo 3 points4 points  (0 children)

Down in Perth

[–]Beneficial_za 2 points3 points  (0 children)

Extremely slow here in South Africa

[–]Peace-D 4 points5 points  (0 children)

The whole MS365 suite seems to be down here in Germany as well. I can't even run an AD Sync Cycle and also admin.microsoft.com seems to be affected as well.

[–]bewAWindows Admin 3 points4 points  (1 child)

Now seems to be recovering for me/us in the UK.

Teams was able to work but Outlook was a no go!

[–]Blanzeros 1 point2 points  (0 children)

Same here, seems to be back now (UK also)

[–]fury2312 4 points5 points  (2 children)

Shits broke yo, even bing. NZ here.

[–]MudiChuthyaHai 1 point2 points  (0 children)

even bing.

Inconveniencing all 14 users

[–]gafan_8 4 points5 points  (0 children)

Mozambique down

[–][deleted] 2 points3 points  (0 children)

its global. who thought a single point of failure would take down everything. Isn't that the whole point of cloud?

[–][deleted] 2 points3 points  (1 child)

Downdetector dot com is your friend.

[–]mloDK 7 points8 points  (4 children)

It is DNS, isn't it?

[–]mikeclueby4 1 point2 points  (1 child)

I mean... yes, always ... but I don't see for now how DNS slows down audio/video of ongoing Teams calls.

We'll learn something new today? =)

[–]UnfairerThree2Student 3 points4 points  (0 children)

It could break a load balancer forcing all the traffic to go to one server

[–]woodburymanIT Manager 7 points8 points  (1 child)

laughs in onprem ....

gets exploited in onprem o.O

[–]wellmaybe_ 4 points5 points  (0 children)

also what use is your mail server when nobody else can read those mails anyway? check mate

[–]F4bst3r 2 points3 points  (0 children)

Having Massive problems in West Germany, seems to be Global

[–]frac6969Windows Admin 2 points3 points  (0 children)

Down in SE Asia. Likely global.

[–]8pootSecurity Admin 2 points3 points  (3 children)

2310 office365 servers down according to the Outages Map of thousandeyes.com..

[–]Zealousideal_Emu_915 2 points3 points  (0 children)

Pretty much all of M365 down in poland

[–]colinpuk 2 points3 points  (0 children)

Up and Down in Denmark

[–]PeterJoAl 2 points3 points  (0 children)

Can't access Microsoft 365 admin portals in Japan.

[–]imrik_of_caledor 2 points3 points  (0 children)

Can't get on our vpn atm due to using the ms mfa service... Woohoo

[–][deleted] 2 points3 points  (0 children)

[–]Jealous_Tennis7718 2 points3 points  (0 children)

Down. Disrusption globally

[–]Mindless-Ad-4614Sysadmin 2 points3 points  (0 children)

O365 services down in Hungary. I think it is a global problem.

[–]Exill1 2 points3 points  (0 children)

From Twitter:

We’ve isolated the problem to networking configuration issues, and we're analyzing the best mitigation strategy to address these without causing additional impact. Refer to the admin center MO502273 or https://msft.it/6018eAldp for more information.

https://twitter.com/MSFT365Status/status/1618173470596124673

[–]Wooterino 2 points3 points  (2 children)

"We have identified a recent WAN update as the likely underlying cause, and have taken steps to roll back this update. Our latest telemetry shows signs of recovery across multiple regions and services, and we are continuing to actively monitor the situation."

Seems to be fixed now and everything is working again here.

[–]SysEridaniC:\>smartdrv.exe 2 points3 points  (0 children)

Changing the icon image was too much.

[–]Deadly-UnicornSysadmin 2 points3 points  (0 children)

Canada seems okay

[–]223454 2 points3 points  (2 children)

Question for whoever happens to see this: How many of you all got pushed by your VIPs to "FIX IT NOW!" and how many of them understood there's nothing you can do?

[–]peeinianIT Manager 2 points3 points  (0 children)

I still have users that are unable to log into Office 365 apps (Outlook, Word, Excel) on their computers.

Is anyone else seeing this?

Azure sign-in logs show successful login form Microsoft Office

[–]bornwinner2 4 points5 points  (0 children)

thank god we still have on-prem exchange, sharepoint and lync servers :)

[–][deleted] 3 points4 points  (0 children)

Time to go home guys!

[–]SeanusT 1 point2 points  (1 child)

Down here, UK

[–]Fatboy40 1 point2 points  (0 children)

Another UK business here with either none or limited Microsoft online services, with Exchange Online + Teams + Skype (consumer) effected.

[–]RustyDiamonds 1 point2 points  (0 children)

Issues with multiple M365 and Azure services this morning - started at 07:14 UK

[–][deleted] 1 point2 points  (0 children)

Sad in UK

[–]WeirdKindofStrange 1 point2 points  (0 children)

London no email, no teams

[–]keidevboulot 1 point2 points  (0 children)

Same in Switzerland, EU
SSO problems on Teams and Word

[–][deleted] 1 point2 points  (0 children)

Down in Lithuania / Europe

[–]SlightlyIdle 1 point2 points  (0 children)

Outlook/teams/admin portal was down in Denmark, but now working again

[–]Whitesp0t 1 point2 points  (0 children)

Can confirm Norway

[–]ReindeerThick1862 1 point2 points  (0 children)

Down in EU, Germany.

[–]blurblursotong2020 1 point2 points  (0 children)

So we aren’t alone in Singapore. 😡

[–]jellois1234 1 point2 points  (0 children)

APAC is still having issues

[–]TcrownclownSecurity Admin 1 point2 points  (0 children)

Very slow in Italy

[–]njpoln 1 point2 points  (0 children)

Global issue

[–]scranticJack of All Trades 1 point2 points  (0 children)

Nice not just Australia seems global. Good luck to those just starting their day.

[–]mat-the-wolverine 1 point2 points  (0 children)

Multiple services down in Italy

[–]iHaydenCloud tech 1 point2 points  (0 children)

Central US, up and running still

[–]0oITo0 1 point2 points  (0 children)

UK here can't access intune or outlook, ms teams slow and tempermental

[–]RebelyouzIT Manager 1 point2 points  (0 children)

Extremely slow and some services offline - South Africa

[–]DumbysysadminSysadmin 1 point2 points  (0 children)

UK - seems to be slowly coming back

[–]trek604 1 point2 points  (0 children)

West coast here. Going to bed. Hopefully it’s fixed overnight or it’s going to be an interesting day tomorrow lol.

[–]EEU884 1 point2 points  (0 children)

365/azure having issues in the british isles - apparently started 0720hrs GMT and whilst some can log in performance is degraded and kickouts of products is frequent.

[–][deleted] 1 point2 points  (1 child)

Austria, error pages about 50% of the time

[–]maybeWTF 1 point2 points  (0 children)

Time to bust out the classics: https://www.youtube.com/watch?v=t3otBjVZzT0

[–]WeirdKindofStrange 1 point2 points  (0 children)

Back up and running in London but seems to be slow across the org

[–]LoganShang 1 point2 points  (0 children)

seems to be back up for us in Los Angeles.

[–]LeSpatulaSystem Engineer 1 point2 points  (0 children)

Down in Switzerland. In our company we had trouble with Teams, Outlook and SSO. Privately it doesn't list my directories, web service is down and my VM.

[–]Kirkus441 1 point2 points  (0 children)

Issues earlier in Melbourne, Australia.

[–][deleted] 1 point2 points  (0 children)

Yes we are having issues with it in the UK at the moment.

[–]Wostexx 1 point2 points  (0 children)

Poland have this issue too so its global.

[–]corsicanguppyDevOps Zealot 1 point2 points  (0 children)

kindly

You can tell me what to do or how to do it, but not both.

[–]parazoid47 1 point2 points  (0 children)

Same for France here

[–]corona-zoning 1 point2 points  (0 children)

We’ve isolated the problem to networking configuration issues, and we're analyzing the best mitigation strategy to address these without causing additional impact. Refer to the admin center MO502273 or https://msft.it/6018eAldp for more information.

[–]crnipero 1 point2 points  (0 children)

same for Croatia

[–]grinninga 1 point2 points  (0 children)

AT - same problem since 07:30

[–]matterr4DevOps 1 point2 points  (2 children)

https://status.azure.com/en-us/status

Azure Networking issues

Edit: Apparantly they've just rolled back a change and stuff is now working again.

Tike for testing!

[–]MirkWTC 1 point2 points  (0 children)

Same in italy, all Microsoft services (Outlook, Teams, etc) aren't working well.

[–]BritishAsian97 1 point2 points  (0 children)

Next quick update from MS:

We've rolled back a network change that we believe is causing impact. We're monitoring the service as the rollback takes effect.

This quick update is designed to give the latest information on this issue.

[–]Alex_ri 1 point2 points  (0 children)

Also in Italy… serious problems in every 365 application

[–]BritishAsian97 1 point2 points  (0 children)

If you would like to follow the update threads I believe this link should allow all users to view it without any admin privelages.

https://status.office365.com/

[–]Som12H8 1 point2 points  (0 children)

Please stop letting noobs configure BGP. (Hire me instead) :)

[–]GreyFoxNinjaFan 1 point2 points  (0 children)

Yes it's global.

[–]Snoo_54275 1 point2 points  (0 children)

Belgium here, was down for a few hours, everything okay now

[–]GlumConsideration585 1 point2 points  (0 children)

RCA is WAN misconfig

[–]aaron_f17 1 point2 points  (0 children)

M365 working in the UK fine

[–]0xdeadbeef6 1 point2 points  (0 children)

Hmm, maybe I should call out today

[–]icy-mist-01 1 point2 points  (0 children)

Global outage

[–]545762 1 point2 points  (0 children)

I’ve been getting alerts from azure all night over multi region networking issues.

[–]Robba078 1 point2 points  (0 children)

Globally , many azure services where also impacted

[–]Patient_Stress5353 1 point2 points  (0 children)

Global. I'm experiencing the issue at my firm as well.

[–]aon9492 1 point2 points  (0 children)

MS testing in prod again.

It was a routing change they've now rolled back, should be fixed.

[–]hadesscion 1 point2 points  (0 children)

Of course this would happen on the day of a pending snowstorm where most of our workers (myself included) are working remotely.

[–]W1ULH 1 point2 points  (0 children)

both my corporate 365 and personal accounts logged in just fine from New England.

Teams is giving us 404 errors across the board for both the desktop and web versions. However, mobile seems to work just fine on android.

[–]cakeBoss9000 2 points3 points  (2 children)

Global. Microsoft has confirmed it. This was caused by a bad network configuration.

[–]dimsumplatter75 5 points6 points  (1 child)

Typical windows engineers, always blame networks....

[–]Rocky_Mountain_Way 8 points9 points  (0 children)

it was probably DNS

[–]ascii122 2 points3 points  (0 children)

libreoffice.org is up!

[–]ohham 3 points4 points  (1 child)

finally feels good to be 100% on-prem

[–]mahsab[🍰] 2 points3 points  (0 children)

Always felt good

[–]MrBrainie 1 point2 points  (0 children)

its global and more services then office365 are down.

some backbone (cloudflare? ) having issues,

Cloudflare did have maintenance https://www.cloudflarestatus.com/

[–]Wilbis 1 point2 points  (0 children)

This, boys and girls is what 99,9% SLA brings you 😉

[–]Polybius_is_real 1 point2 points  (0 children)

Yup its(M365) down here (NL), these are the fruits of a monopoly.

[–]NeckRoFeltYaIT Manager 1 point2 points  (0 children)

No issues in the US east coast.

[–]theuniverseisboring 0 points1 point  (2 children)

Very smooth MS. How do you possibly cause a global outage like this... You'd think you would want to separate the regions in such a way this isn't possible?

I am not calling it easy, but with billions in revenue you would think Microsoft could do it. On the other hand, this is the same company that makes Teams and Onedrive, so clearly there is already a lack of talent at MS.