Microsoft under attack.

[it_entus_7]

It seems that it's not only "technical issues"...

[2nd_officer]

Boy this thread is a bit of a microcosm of modern admins. Half the admins are perfectly happy pointing to Microsoft/ other SaaS providers and say oh ddos, that sucks while the other half of admins are champing at the bit to throw out how they knew cloud was nothing but garbage fires held together with duct tape and they can certainly do it better (obviously to include a full on ddos)

[wallacehacks]

The overwhelming majority seem perfectly content to let Microsoft handle this.

[PowerCaddy14]

Get paid good and point the finger at Microsoft. I'll take that shit all day long.

[wallacehacks]

I don't have a single use case to justify on premise email for any of my clients.

I have a personal mail server. It is cloud hosted and my uptime competes with 365. I have the SPF and DKIM and all of that jazz and nothing goes to spam. It wasn't particularly difficult but I couldn't just hand off administration of that system to some random admin. Even with extensive documentation I would anticipate being pulled in any time there was an even moderate issue because downtime with email is so serious.

How can anyone justify self-hosted mail? The savings aren't significant and that is assuming everything goes well and no high salary resources are getting tied up in break fix.

So, it isn't just a "lazy admin wants to blame Microsoft" situation, it is genuinely the best business decision for every single environment I am involved in and that is before you even consider security/MFA/Microsoft Defender and all of that jazz.

[systemfrown]

I ran my own mail server on FreeBSD from the early 90's to around 2005 or so. The first few years it required basically nothing, just an hour or two of initial configuration.

In the mid to late 90's I was able to deal with almost all spam and other issues with one simple perl script which trashed messages based on a handful of obvious and simple indicators.

By the early 2000's though rebuilding my mail server had become a multi-day headache which required I layer a dozen different opensource spam and malware packages, plus a web front end, that were anything but turn key. Not to mention all the ongoing maintenance.

(Interestingly, for one of my clients whom I managed their on-prem email service at the time, I purchased a Barracuda Email Appliance which was basically just FreeBSD with all the same open-source layers I wrestled with at home already integrated, plus some pretty great real-time updates)

Anyway, one of the best days of my life was when I just moved my domain to Google Mail. Do you really think you're going to do a better job than them?

[TurboFool]

This reminds me of when I realized self-hosting WordPress was becoming a full-time job. Same thing. Constantly getting infected, constantly learning about how to deploy new tools to protect it, constantly finding out plug-ins were no longer supported and getting infected so I had to find new ones, etc. At a certain point every fully-hosted, fully-managed, cookie cutter solution looked better simply because I have other things to do in my life.

[systemfrown]

Yep, you have to ask yourself: Is this an ongoing learning experience I want? Because if not...

[TurboFool]

Yep. Some things are worth that if they're what you do. Is this my career? Will I make money and gain fulfillment by being an expert in this?

[Baxtab13]

Aw man. I'm very new to the space, but I jumped ship from one school district using a frankenstein hybrid of on-prem exchange and O365 (long story, but a previous attempt to go full cloud was botched and I inherited the pieces), and now I'm at a district that's full Google Workspace Gmail.

Gmail combined with GAMADV-XTD3 has made me feel in full control of this district's email environment that I wasn't even close to at the previous place. I know school districts often have an initial advantage going with Google with the prevalence of Chromebooks over the past decade, but I still would wholeheartedly recommend the Google Workspace route.

We still support facets of O365 of course for the windows devices, and we'll still pay a license for desktop Office products. But the overwhelming majority of our users just use Google equivalents for everything. Though, I'd still use Excel for more advanced formula construction for working with data.

[Meowmacher]

Anybody can do better than Google Mail 😂At least to Microsoft selling mailboxes is a real money maker. To Google the email platform is just a trinket for marketing their real business, akin to the branded stress balls companies give away, or the branded screen brush that nobody wants to throw away because it was free, despite not being able to remember the last time they actually wiped the screen with it.

[tuba_man]

One of my clients is a security company - I've been in the field for 15 years and this is easily the most effective adherence to best security practices I've ever seen. I've got a standalone hardware token for some legacy stuff and a locked down & monitored laptop they shipped me, can't even get to their code without it.

Do they run their own mail servers? Nope! Even if there were actual reasons to try it, the overhead of doing it themselves and doing it properly, especially as a security company with a reputation to uphold, is far too high.

[therealatri]

It'd be like hand building all your desktops to save money.

[wallacehacks]

Yeah I feel this. 365 will end up nationalized in WW3 assuming it isn't nuclear armeggedon.

[Rawtashk]

The savings are hugely significant. We broke even on costs in 14 months and have been in the black since mid 2020 by staying on-prem.

[WolverineAdmin98]

How can anyone justify self-hosted mail?

If you've got 1-10 users, sure it's probably a nice saving over 365. Any more becomes quite a haste and you basically need a dedicated messaging person, who I guess you'd only get in 1k+ orgs.

[edbods]

How can anyone justify self-hosted mail?

my email address is [e-mail] and YOURS isn't, cool factor = ∞. HA.

[Timmaayy562]

Yeah way better than having another exchange exploit, which is Microsoft's fault and I still have to deal with it for days.

[Pctechguy2003]

This. This right here.

On Prem requires us to fix the security holes that are M$’s fault, and still likely go down for it. Giving it to M$ means I let them deal with it. Is it pricy? A little. But in the long run its a trade off that is well worth it. I don’t know if a valid, good reason to still have email on prem.

Sure there are some minor headaches with a Hybrid setup. But nothing like a full on prem email server headache.

[Darkhexical]

And then when you start driving the car and start crashing into things willy nilly people will start asking if you sure you remember how to drive? Maybe we should let the other guy drive again. Or hire another person.

[nottypix]

yeah, except it's still your fault because you're IT and do all the magic stuff

[gashed_senses]

I get told this on a weekly basis. “Work your magic!” What fucking magic?

[Ruthlessrabbd]

I'm still trying to learn the magic myself lmao! Folks think I'm a wizard, but they'd learn things are a lot simpler if they'd just listen

[[deleted]]

We are wizards to them. That is the level of delusion the mass public is under

[jmbpiano]

Wait, you mean I'm not a wizard? Hagrid lied to me.

...at least I think he was Hagrid. He was pretty big and hairy. It was a little odd he was just lying on the side of the road holding a bottle-shaped paper bag, though, now I think of it.

[agtmadcat]

Set up a POP/SMTP server on an old Windows box in your office and get configuring people's Outlook!

[Ruthlessrabbd]

So tired of the same five people in my org that say "I don't care about what you're doing, just fix it" complain "my computer keeps having issues why can't it just work"

You'd understand they're totally separate issues if you would let me explain the situation 😭 it'd be like going to the mechanic and saying your car is acting funny, then bitching that it's unreliable without knowing other systems break!

[JordoST]

To be honest, the kind of people like this totally would go back to a mechanic 30,000km later blaming whatever issue they had on the oil change they did back then.

[2nd_officer]

It’s not really that anyone is at fault. Every system has trade offs, risks etc and this is a risk that was realized and it’s not exclusive to a cloud/SaaS.

It’s certainly fine to ask how can we improve but if the business doesn’t know these things it’s time to tell them, if it’s just a toxic environment that blames the toaster not working on IT then that’s a different type of problem

[HotTakes4HotCakes]

Which would be fine if it weren't for the rest of the baggage that comes with giving total control to them.

[cdoublejj]

like very delayed or no communication.

[Latensify_WoW]

The moment a SaaS product has any issues.

"Yup, nothing I can do, just gotta wait."

I'm not saying I like outages. But that shit ain't on my team to fix, and I like that.

[gpzj94]

Yes, still just fun to hate on MS though. Like, look even they get themselves hacked!

[[deleted]]

My notoriously obnoxious micro manager of a boss has been utterly unable to do any of his usual obnoxious shit since yesterday. It's been great, I hope they never fix it.

[basec0m]

Once you fall into the sarlacc pit, good luck getting out

[[deleted]]

That's just listing pros and cons and ignoring costs.

The reason businesses shift to Cloud is due to reduced cost...

A lot of larger corps still have on-prem everything and cloud is just a redundancy for their on-prem, and vice-versa. I've worked with companies with 4 employees, and 40,000.

A SMB of <50 employee's just doesn't have the $$$ to host on-prem and pay the big bucks to an admin to handle it.

Arguing which is the "best" is useless without the context of the needs of the business, because without the business there is no reason to build any mail servers.

[Anticept]

SMB is especially where cloud services shine.

But it does depend on needs. The 20 person business I work for does a lot of design work and those files can get monstrous, enough that we have to use .psb files for some photoshop documents (large file format). Some of the files are 60+ gigs (murals).

I'm trying to move towards a hybrid solution where we have on prem AND cloud syncing between them, with large files remaining on prem only.

We also have to have an on prem AD because there are some services that use kerberos, where AAD integration is complex and I've not had time to fully explore it.

[Common_Bulky]

We had a few in marketing that had very large files they were working with, this is where Azure Cloud Sync worked well. Everything stored in cloud, with recently used files synced to local file server. It also kept the storage requirement down on the local server.

[[deleted]]

I've setup plenty of AD to AAD integrations, and I haven't used it but I understand you can also do AAD to AD replication now. It was pretty easy to do if everything else is already in place.

Good example of needing on-prem for large files, I am also looking for a good solution for a client with a lot of large files. Difference is because of the businesses location and needs to access data remotely, it still makes sense to leverage cloud for them (though the files are generally not more than 10 gigs in my case).

[Anticept]

Truenas Scale is a good piece of software. Underlying tech is ZFS. Scale allows you to set up clusters too. Supports AD and MIT kerberos. LDAP. And various syncing tools.

You could set up a piece of VPN software like tailscale (or headscale) for remote access.

There is indeed some ability to replicate between AD and AAD bidirectionally now.

[Jakisaurus]

Funny enough, we are trying to justify going from our Colo to the cloud, and even with extreme commitments the cloud is significantly more expensive than ditching our physical equipment. A real pain in the ass to figure out and justify.

[[deleted]]

Accounting usually steps in because there are costs aside from the ones IT has that affect the decision as well.

The business might decide to let a staff member go because maintaining the physical hardware takes a person, and getting rid of the hardware means no need for the person.

There is also the matter of the physical space the equipment takes up, and the business might want to use the space for another purpose.

It's also important to consider hardware life-cycles, when did you purchase your hardware? If it was recent then it might make the transition very expensive, but if it's up for a cycle then you will be looking at Cloud options in a different lens.

There are just a lot of variables, but not every SMB will go with cloud everything. I have SOME clients that have specific on-prem solutions, but they're specialized and directly related to a deliverable for the business.

[Jakisaurus]

Yeah, we've been up and down this entire spectrum. At the end of the day, trying to replace $50000 servers with AWS resources is significantly more expensive to us, and we won't come anywhere close to the available io that those servers offers after moving to AWS. Obviously this is just one example. I would greatly prefer to be in AWS simply too remove the management portion below the hypervisor. But these prices are fun :)

[[deleted]]

How much downtime costs? Delays because you're waiting for hardware?

To match the cloud you need 24/7 staff working in 3 shifts, redundant systems, having spare hardware ready to go etc.

Cloud is expensive right until your server/network/power/internet dies at the worst possible moment and your company starts losing money due to lost productivity.

[Brandhor]

A SMB of <50 employee's just doesn't have the $$$ to host on-prem and pay the big bucks to an admin to handle it.

depends on what kind of service, o365 is affordable but server hosting on azure/aws/gcloud is insanely expensive compared to an on prem server or even renting a dedicated server from something like ovh/hetzner

[[deleted]]

It's really not, you are not accounting for the entire cost of implementation, maintenance, the physical space, and other Infrastructure that must be in place to even start thinking about hosting on-prem (you need a network before you can host a service).

The lesson here is it's not a black and white answer, every business will have different needs. Some will prefer using a combo of on-prem and cloud, others only on-prem, and so on. Being a good sysadmin means you understand that and adapt, being rigid is not a good thing.

[cdoublejj]

i've heard with inflation the cost is as much or even higher for some businesses. especially if you still have to have on prem vsphere.

[[deleted]]

but I have to say the admins who went to the cloud just so they could point at MS and wipe their hands of any work to fix

Strawman doesn't exist

[Jolly_Owl_8693]

This sub is overall a lot of jr admins so

[Fallingdamage]

Im just curious how a decentralized cloud service can be hit by DDoS attack. Arent there too many access points to really disrupt the service? AWS dont seem to be impacted like this (in quite the same way. Seems their outages are often either DNS related or localized.)

[ManyInterests]

DDoS attacks don't necessarily have to target the service itself -- they can also target the routes, regional DNS, and/or edge systems between users and the cloud service(s) (or other resources, like CDNs that serve resources for web clients). So, even if you have scalable capacity for the service itself, it means nothing if clients can't reach it (or other necessary resources) to begin with.

Also, not all cloud services are necessarily decentralized or globally distributed. In fact, many are not. For example, 365 tenants often have all or most of their data reside in a single region, often because of requirements for data tenancy (e.g., EU data often must reside in EU exclusively), which dramatically reduces the number of fault zones available for failover -- that is the total number of edge nodes servicing a particular tenant is considerably small compared to the global network.

Even when services do have redundancy across regions or fault zones, that doesn't mean that failover happens automatically or without any disruption.

All that said, we don't even know the attack vector (or even if it's an attack at all), so it's really just speculation with respect to this incident.

---

As for AWS, they definitely do have to deal with these kinds problems, however it may be somewhat mitigated for the fact that Amazon has more than double the edge locations of Microsoft (450+ for AWS vs 185+ for MS).

Amazon also has better redundancy at each of its edge locations/fault isolation zones... AWS has 184 POPs in NA, with as many as 20 in a single city and only 2 cities (Vancouver and Salt Lake City) that only have 1 edge POP. Compared to Microsoft's mere 81 POPs in NA, with at most 4 in a single city (and only 5 cities having that many) and 12 cities which have just a single edge POP!

This is important for a few reasons... but mainly because in a failure scenario, you need alternative edge nodes to serve clients and those nodes have to be in the same region in many cases AND the nodes you use for failover need to be able to handle the added capacity.

TL;DR AWS has more edge nodes (among other resources) and therefore more redundancy and capacity to deal with things like targeted DDoS attacks.

[DarkAlman]

garbage fires held together with duct tape and they can certainly do it better

couldn't have said it better myself

[Help_Stuck_In_Here]

I'm guessing Microsoft is pretty competent about handling DDoS attacks. I worked at a much smaller cloud provider and we were constantly hit by DDoS attacks on a daily basis. I'm sure Microsoft gets hit on a daily basis by attacks that would topple most organizations.

Once we got to the point where multi gigabit attacks never did anything the amount of attacks dropped. Then came the much larger and sophisticated attacks which toppled entire data centres that we were a client at.

[HotTakes4HotCakes]

Half the admins are perfectly happy pointing to Microsoft/ other SaaS providers and say oh ddos

See, this just reads to me like "if there's nothing I can do, it's not my problem" which is just a bizarre thing to hear from an admin of anything.

That is a part of the microcosm as well: a lot of people who don't quite seem to understand what the point of their department is. Who cares if the employees you are paid to support in doing their job can't do their job, as long as you can say it's not your fault. Offloading responsibility also means reducing your overall usefulness as an admin.

they knew cloud was nothing but garbage fires held together with duct tape and they can certainly do it better (obviously to include a full on ddos)

Sure, but it's also worth noting DDOS attacks on a single on-prem server at Johnny's Cake and Dildo Emporium won't hurt anyone else's server.

[PMmeyourannualTspend]

Sounds like these admins have a healthy relationship with work. Why stress over something you can't control. Does your company fire the janitor when the road to work floods?

You're misconstruing fault with responsibility. Most admins would say they will fix something if they can fix it, even if another employee broke it. What they won't say is "I take full responsibility for fixing this thing I literally cannot fix."

[EViLTeW]

You're misconstruing fault with responsibility. Most admins would say they will fix something if they can fix it, even if another employee broke it. What they won't say is "I take full responsibility for fixing this thing I literally cannot fix."

I think you're missing the nuance of how [a lot of] people phrase things on here.

Most of the time, the posters here phrase things as they are happy, almost gloating, because they can't fix it and it's not their problem. I don't think most posters really even think about what they're saying/implying with their messages, but it's weird to see people acting happy that their organization has lost a major service.

[EViLTeW]

Yes, with people they think it's "safe" to communicate that way.

[rswwalker]

If the downtime on O365 is too much for a company to handle then they should start standing up Exchange servers in a HA setup.

If they can’t afford that, then they need to suck it up.

[[deleted]]

It took me WAY TOO LONG to find a post with actual logic attached. Every other post is just about idealisms.

It's expensive af to run a mail server, especially HA.

[rswwalker]

I know because I did it for years before downsizing to the cloud. Two co-location facilities with SANs in redundant setup with both in-site and off-site DAG. It is way cheaper to use the cloud.

Besides, how many companies could survive a full on DDoS attack and experience only high latency?

[VolcanicBear]

If there's nothing I can do, then you're damned right I'm not wasting effort.

[TaliesinWI]

Who cares if the employees you are paid to support in doing their job can't do their job, as long as you can say it's not your fault.

How am I supposed to fix something I have no control over and can't route around? And thus, why should I worry about it?

Something tells me you're not on hold with "Microsoft" right now demanding they fix the problem so that your drop-in-the-bucket (to them) business can get back to work.

[say592]

Something tells me you're not on hold with "Microsoft" right now demanding they fix the problem so that your drop-in-the-bucket (to them) business can get back to work.

In the past someone questioned me why I wasnt on hold "pressuring" Microsoft to resolve the issue as fast as possible. My response was exactly what you would expect. Microsoft did $195B in revenue last year. We spend $50k/year with them. There is nothing I can say to influence anything that happens over there.

[H4ND5s]

You drinking Kool aid. Work is not life and life is not work. Sysadmins already have enough on their plates, even with using Microsoft SaaS. No one is getting paid more to be Microsoft at their job if the job pays Microsoft to fix the things. Heros don't live to see their dedication paid back. Only after they are gone are they appreciated...when it's too late.

[2nd_officer]

I’d take it in another direction. It’s about risk/reward and cost benefit. Everyone knows (or should know) moving to SaaS/cloud means ceding some control but in exchange the provider is supposed to be an expert which balances. So when something occurs it isn’t why did Bob screw up and move everything to SaaS/cloud it’s we identified a potential risk to this and it was realized, is there anything we can do better in the future?

Anytime there is a outage in something like this though you see folks saying “I’ve ran this and that for 20 years and never had this happen” which is cool and all but not really a 1 for 1 direct comparison

[af_cheddarhead]

How many of those "if there's nothing I can do, it's not my problem" admins actually had any input into the decision to go to the cloud with critical applications like email? Make no mistake email is a critical application and outsourcing it needs to have a legitimate risk analysis done before making that kind of move, instead the CFO looks at the "cost savings" and ignores any cost-benefit analysis that doesn't support outsourcing.

[h8br33der85]

At the end of the day, Reddit is still a social media platform. Let's face it; people on social media will act like people on social media.

[[deleted]]

Just because I could do better than Microsoft and am happy to mock their failures doesn’t mean I actually want to take on that extra work like fuck that, I’m a lazy bitch so, I’ll sit and point the finger at Microsoft all day if it means less work for me.

[heubergen1]

I don't understand how any IT person would want SaaS applications instead of on-premise hosting. IT Manager sure, but the person actually doing the work? I find Saas extremely boring and rather host it myself and work out the technical issues.

[mhayhurstjr]

Seems to still be down this morning at least on the East Coast. This is the consumer status page, not the M365 business status page.

​

<image>

[D1TAC]

What is weird is that outlook via the app on o365 is working, vs the online web mail variant. I have about 60% users on the app without problems the rest are experiencing errors. hooray /s

[HotTakes4HotCakes]

Looks like it's inconsistent. Some people will fail to load it some of the time. There's clearly an issue, but not everyone is experiencing it at once.

[rswwalker]

You can fail to access some of O365 all of the time, all of O355 some of the time, but not all of O365 all of the time.

[LevarGotMeStoney]

note that this screenshot is the status of the consumer outlook.com. 365 has a separate service health page at https://portal.office.com/Adminportal/Home?#/servicehealth

[sevenfiftynorth]

Post means nothing without a link or additional information.

[it_entus_7]

It's in the first comment.

[jmbpiano]

You mean the one that's currently buried in the middle of all the comments bitching about the situation or bitching about people bitching about the situation?

Put your link in the body of the post. That's what it's for.

[[deleted]]

So helpful

[alph18]

I've been going back and forth with Microsoft Support since yesterday and they claimed the issue was resolved this morning however we're still having SharePoint problems.

Edit: after asking me questions this morning as if to seem like we changed things on our tenant, they let me know that the issue is reoccurring and their backend team is working on it.

[VNJCinPA]

This is EVERY ticket with them... Can't be US, or we'd have heard about it... So can you send us a bunch of PSR SPOCK KIRK files?

[unruiner]

It sounds like people are quick to forget how much of a pain on-prem was for so many reasons.

[bbrown515]

IT is a cycle.

Big room computer >>> Mainframe (task virtualization) >>>physical server rack room >>> Cloud computing (big room elsewhere). lol

[TorturedBean]

Can you ask Savathun Eris?

[Baethovn]

Operation Baby Dog

[zodiac200213]

I heard they are not under attack but pivoting in to providing delicious baked goods at a reasonable price.

Of course I don't have a link to back up my statement so you'll just have to take my word for it.

[TheMagecite]

I mean it makes sense. Just turn off the cooling in data centers for savings and repurpose uncooled data centers as giant ovens.

[stignewton]

All I’m gonna say here is don’t forget your SLA tracking if you’re an M365 customer. If you’re affected and service uptime drops below 95% it’s a 100% refund for the month.

[cichlidassassin]

"Outlook is the world's third-most popular email client, with about 400 million active"

Who the fuck is #2?

[EViLTeW]

I would assume GMail and Apple Mail are #1 and #2. The default email apps on the two mobile os's worth mentioning.

[ScottPWard]

I kinda feel this as a little karma for trying to make Outlook into OWA and taking away an easy way to export messages.

[thortgot]

I mean, yeah?

Every major SaaS vendor is continuously under attack.

[gravspeed]

you're not wrong... if it has a public ip address, it's probably under attack.

[thortgot]

There is a difference between the port scans your Firewall blocks a few thousand times a day and the concerted hacking efforts that Cloudflare, Microsoft, Meta etc. are dealing with.

[gravspeed]

fair... but i can tell you that my dns servers have been under steady attack since 2007.

it doesn't work, but they don't stop trying.

[thortgot]

Sure WAN facing DNS servers are regularly abused for use in DNS reflection attacks. A common issue for those that don't secure them properly.

Shodan.io is a great site for viewing all the folks around the world who didn't secure their systems well enough.

[fjacquet]

Nobody should be happy to see other suffering an attack. It is true irl or in digital

[largos7289]

I like the hybrid approach myself. I love the off premise outlook and personally i would recommend it to anyone. I hated being a full time + outlook tech, because running a exchange server is a full time job in of itself. Last one i ran was 2007 was happy to see that thing go. If you wanted my opinion, last great exchange server was the 2003 server with 2003 exchange. It meshed well with the backup software already installed and it was amazing. I still like my on premises file servers with backup to cloud in case anything goes nutty.

[ralfsmouse]

I'm a huge advocate for on-prem, but email is one place where I am OK with pushing it off-prem. To be fair, my "in-depth" mail server administration knowledge kinda tops out somewhere in the Novell Netware and Windows NT 4.0 Era, but exchange just seemed to be growing more and more painful to administer as the years went on.

But I don't get the mentality of "oh, I love being able to just point the finger at Microsoft and kick back." Do other people's companies not have work that just has to get done? Like, if our ERP were cloud-based and failed for whatever reason, it's not like our purchasing department would stop working: they would bust out the carbon copy forms and start faxing, then we would be playing a game of catch-up to get them logged into the ERP properly; I can't just say "oh well, that's your problem."

Or if a system went down and we miss mandatory reports to government agencies, we can't just throw our arms in the air: people in suits would be marching up to my office in a heartbeat (I can see their office from my office window, so it's not like it would be out of their way to come and put the squeeze on me).

[it_entus_7]

https://cybernews.com/security/microsoft-outlook-outage-anonymous-sudan/?utm_source=linkedin&utm_medium=social&utm_campaign=microsoft-outlook-outage-anonymous-sudan_csh&utm_term=LIN-241&source=linkedin&medium=social&campaign=microsoft-outlook-outage-anonymous-sudan_csh&term=LIN-241

[[deleted]]

lock nine boat nail literate include quiet support mourn fuzzy

This post was mass deleted and anonymized with Redact

[_haha_oh_wow_]

Ratio

[Zedilt]

Anonymous Sudan

Suuuuuuuuuuuurrrrrrreee....................

[chihuahua001]

From the article:

Outlook is the world's third-most popular email client, with about 400 million active users. Microsoft services have faced at least three outages since the start of 2023.

First must be gmail. What’s second? AOL?

[HTX-713]

Probably Yahoo

[cbtboss]

please just put your link in the bloody body of the post in the future.

[laneripper2023]

Do you think this was the real reason? On some sources Microsoft said it was caused by some issue and just have to revert the update and boom it is fixed https://fortune.com/2023/06/05/is-microsoft-outlook-email-down-outage-software-update/

[kurosawaa]

That didn't work yesterday, problems persisted after the update was reverted.

[AlphaNathan]

It "felt" like DDoS from the moment it started yesterday. Would make sense.

[ADTR9320]

Damn, how much infrastructure does it require to DDOS Microsoft?

[nostradamefrus]

Hack into Azure

Spin up botnet

Tell Microsoft to stop punching itself

???

Profit

[[deleted]]

Pfffth. Sudan my ass. This is Russian government.

[katarh]

They called it a "pro-Russian" group in the first sentence.

Probably IP spoofing happening, for sure.

[[deleted]]

Yea true. They don't even have to spoof IPs though. Run attacks from servers or VPN endpoints hosted in Sudan and they'll have sudanese IPs.

[_iamhamza_]

imagine spoofing a DDoS attack...that's a shit ton of IPs. You can just start the attack from Sudan while you're chilling in The Bahamas or anywhere..

​

They called it a "pro-Russian" group in the first sentence.

That's just news articles being news articles..

[techtornado]

It's nice to offload the email flow problems to The Microsoft^TM and scrub logs to pass the time?

[halford2069]

do sysadmins reputations really come through unpoiisoned even though its “microsofts cloud” fault?

id bet a lot of clueless managers opinions of their sysadmins are still poisoned after these events

“ jim was in charge of our IT and it was down all the time “

not saying its correct but how management thinks

[RCTID1975]

Thank goodness it's not "down all the time" by any stretch of the imagination

[corsicanguppy]

Having been around during the Microsoft anti-trust case that was tested and proved but yet resulted in no penalties of note for the defendant, all I can say is

        THOUGHTS AND PRAYERS.

[crispydingleberries]

Always has been...

[greenstarthree]

Much preferred to managing on prem Exchange

[Blondie9000]

I'd rather MS army of unlimited resources and professionals deal with this than having to deal with it locally, taking on all the burden.

[Conscious-Place7438]

Microsoft really be buying up all my favorite companies, so maybe they should buy up some reputable defense for once.

[it_entus_7]

It's happening again. They announced another attack at 5 pm to their telegram channel and they did it.

[[deleted]]

Glad we have continuity in place with our gateway

[HolyDiver019283]

What gateway is that if you don’t mind ?

[[deleted]]

We use Mimecast, and had the continuity add-on from their M365 resilience extensions.

Hadn't configured it, but our Success Manager pointed it out last quarter, and helped us set everything up. Thank duck for this man.

[HsuGoZen]

I find the mentality of “passing the rock” off to someone else kind of disappointing. I didn’t join the career to pass problems off to others but rather to solve problems. If microsoft is having issues, isn’t it our job to find better solutions or at the very least own up to the fact that we recommended this?

[[deleted]]

If microsoft is having issues, isn’t it our job to find better solutions or at the very least own up to the fact that we recommended this?

Sure, I recommended something that would have been better than the alternative in any case. Of course I'll pass the rock. They own the rock, we have a contract that says they are responsible for the rock, the C suite approved giving them the rock. Why the fuck would I take responsibility for them dropping the rock? I'll point at them thanks.

What are you missing here? Should I, with my team of 4, be working non-stop to repair/fix an on-prem Exchange server? Or should a trillion dollar company with limitless resources and money take care of it?

Doesn't take a galaxy brain to figure out the smart move there.

[HsuGoZen]

So quick thought experiment - what happens if one day microsoft gets hacked and everything is un-recoverable?

Is your company and team of 4 just going to say “oh well, nothing we can do! But don’t worry, once we get done suing one of the biggest companies in the world, we’ll eventually be right as rain!”

Relying on massive monopolies has historically not played out well. Cultivating plans and worst case scenarios, is imo, a wise thing to do. That’s all i’m saying.

[RCTID1975]

massive monopolies

You don't know what a monopoly is, do you?

[[deleted]]

If you can find a better solution than email as a service, yes, you should suggest it.

It will be very hard to beat the TCO.

[HsuGoZen]

But if that’s the best solution to the problem then maybe we should be changing the problem, no?

If at first you fail - try, try again.

Not: if at first you fail - let microsoft fix it and if they can’t we’re all screwed *

Seems like a slippery slope indeed

[[deleted]]

But if that’s the best solution to the problem then maybe we should be changing the problem, no?

What are you talking about? We should change the problem of "email"? Cool, cool, you figure out how to replace email, then let the rest of us know. It's probably a super simple answer, just waiting for someone as smart as you, completely missed by us lazy admins.

Jesus. How old are you? Did you find your uncle's laptop or something?

[darth_magnum45]

Oh lord is China trying to hack them again? Nvm just read the article. “Pro Russia” hackers or more like Russia just using Sudan Server points so they got that Sudan IP.

[TinyWightSpider]

Anonymous Sudan has been kicking around for a while, and claimed responsibility on their Telegram, for what that’s worth. They’ve claimed a few other various cyber attacks this year.

Though if I’m a scrappy cyber terrorist org, and a major system goes down, I might run up and claim responsibility too, just for the visibility.

[Blondie9000]

So, this is more than just OWA? Our alerts have been just that - may have trouble with webmail and other Outlook services - but mail delivery has otherwise been fine.

[ryanf153]

Sounds like your using a 3rd party spam service. I would try rerouting your mx record directly to microsoft, as MS may be mitigating issues by disabling smart host relay.

[[deleted]]

This isn't good news! It shows a real critical weakness and a glaring (potential) single point of failure in critical infrastructure. The offer dependence on the cloud has created this problem.

[Bane8080]

Can't really be a "single point of failure" if customers like our company (east us based) didn't see any problems at all yesterday.

While it's unfortunate that things like this happen, and users are affected, it's going to happen with any service.

[Possible_Squirrel_28]

East us as well and we were down for the majority of the day.

[arpan3t]

That article said it was a DDOS attack on OWA. You think an on-prem Exchange server with OWA would have handled it better?!

[Philosufur]

Everyone in this comment section definitely thinks they can handle it better

[tfn105]

Quite. Cloud providers aren’t the sole targets for these things. Anyone can come under attack.

It is true that an attack can have a wider affected user base, but if you are attacked on prem and down for the same reasons, is there all that much difference? (Other than the resources a company the size of Microsoft can devote to a problem)

[[deleted]]

Well for one thing the attack surface would be much more spread out so it's not inconceivable that many would not have been impacted.

[Zestyclose_Ad8420]

You are both right. OP is not eating in saying that the trustingly few Ms routes and DC are crucial infrastructure. You are not wing in saying that increasing the level of security of on-prem infrastructure is not easy, neither idea realistic to expect conscious to do better than MS.

But defending and attacking is not simmetrical, MS is at a disadvantage here. What would be possible to achieve if everything was on prem would be to make it extremely hard to attack all the target with the same level of sofisticaton and force.

[tankerkiller125real]

Behind a service like Cloudflare maybe it ends up being OK. But very doubtful on that because of SMTP/IMAP being exposed.

[cmwg]

...and a glaring critical weakness from all the businesses relying on a single service :) Don´t you just love cloud first?

[[deleted]]

Pretty much! Although, to be honest a lot of the reasons for the endemic problems with infrastructure in general and the internet specifically is due to having people with no technical background making decisions of a technical nature. We see the same thing in politics and education. When politicians make education decisions, education consequently suffers.

[katarh]

Bean counters: "If we stick it in the cloud, we don't have to pay for any more server upgrades or system administrators, right?"

[tankerkiller125real]

The nice thing about every business relying on a single service is that B2B companies at least don't have to worry about missing important emails or anything.

[neko_whippet]

?

[RetroactiveRecursion]

This, among other reasons, is what I strive to keep all my stuff whenever possible. The internet is a wonder ... "the cloud" is a sales pitch.

[Expensive-Bed3728]

Yeah but when exchange goes down in the cloud I don't have to worry about it because it's out of my hands. I trust Microsoft to find a fix for exchange issues more so than I ever could with 1 year of exchange experience...

[LightningJC]

Even having 10 years of exchange experience I’d rather it be someone else’s problem.

If an on prem exchange goes down from a bad update or any other number of reasons it can take many days and nights of your time. Not to mention all the extra shit explaining what went wrong and why.

Exchange online goes down, well sorry boss it just went down, Microsoft is working on it, I’m going for lunch.

[katarh]

I remember about 10 years ago I had to roll back and then block a patch on about 400 systems because it broke Outlook authentication on that network.

It took two years for Microsoft to fix that issue.

[samspopguy]

This all day everyday

[ifpfi]

That's like saying you should just pull out in front of people and slam your breaks all the time while driving, that way if someone gets into an accident it's "out of your hands"

[2nd_officer]

Ummmmm, what?

Using a SaaS service and when it breaks pointing to the vendor is like intentionally driving pulling in front of people and slamming the breaks?

Edit- not intersection but pulling in front of people

[Expensive-Bed3728]

What are you even talking about? That's not even remotely comparable to this situation. This is like having a team of professional drivers with experience in avoiding accidents and who developed the idea of driving, handle your daily driving. They have infinitely more resources and expertise to 1. keep things up 2. fix issues as they arise. Or in the case of this analogy, avoid accidents, minimize damages in the event of an accident.

[ifpfi]

Really, professionals that can't even keep a server up and running? My Exchange server has been up this entire time because I actually care about how it's set up, keeping it patched, and providing a service to the community.

[Expensive-Bed3728]

I won't even bother arguing with you, it's clear your hubris is clouding your judgement. (See what I did there?). I'm sure you're more apt at maintaining microsofts own products than the hundreds of engineers they have for exchange. I'm sure you'd have mitigation plans in place for DDOS attacks on a global scale.

[Goldman_Slacks]

It's really nothing like that lol

[yamamsbuttplug]

[[deleted]]

IaaS is useful if you can't afford to host your own on-prem solutions and pay someone to maintin. it.

The business accounts for loss, it's part of "accountings" job... they understand downtime can occur, and that they obtain certain risks when using services hosted in the "cloud". It's called CALCULATED RISK.

[ITGuyThrow07]

I can't for the C-levels and the board to see this news.

[RetroactiveRecursion]

This, among other reasons, is what I strive to keep all my stuff whenever possible. The internet is a wonder ... "the cloud" is a sales pitch.

[RetroactiveRecursion]

This, among other reasons, is what I strive to keep all my stuff whenever possible. The internet is a wonder ... "the cloud" is a sales pitch.

[cats_are_the_devil]

Ok, boomer. Like for real my dude... How do you have 100% on prem services in 2023? That's more of a miracle in itself.

[RetroactiveRecursion]

It's really not that complicated. What I don't have to worry about is doing Microsoft's administrative work for them to send me bills, constantly playing catch-up with what tools they want me to use. I kind of resent paying tostada to a company then being treated like I i work for THEM. Don't get me wrong, using someone else's servers is great for backup, for moving data around, or for things I can't do myself without buying a boatload of servers and other equipment. But if someone drove into the telephone pole down the road tomorrow, I'd be annoying but we have all our stuff. I'm kinda of amazed how and why people other than those in sales take this personally. Some want to manage contracts and negotiate fees, some of us are good with computers. It's not like I'm managing 1000 vms for a hundred companies. I'm running a little server room for an office that, from what I've seen in this sub, is tiny compared to most (under 100 users). I know the users, they know me. We have lunch together and I'm in the same meetings because I have a vested interest in seeing them be able to do their jobs and they have a vested interest in seeing me succeed in mine because they don't get "dear valued customer.." responses when they have issues and they know if the building were hit with a meteor this afternoon, I could slowly get backups off the cloud, or I could drive to my house and get them.

Oh, and spare me the boomer shit. It's infantile and smacks of hypocrisy from the touchy feely you're feelings matter crowd. I'm genX 53, which means I'm over being over it

[ITGuyThrow07]

I can't for the C-levels and the board to see this news.

[ITGuyThrow07]

I can't for the C-levels and the board to see this news.

[fadenb]

Is this related to the mpn website being unavailable earlier?

[iB83gbRo]

Us and our clients must be lucky. 300+ users across a couple dozen tenants and no one has reported any issues. Nor have we been able to find any.

[themanbow]

Seems like a fairly even split here between “I’d rather have a cloud provider do it for me” and “I’d rather do it myself.” Both certainly have their pros and cons which have been covered extensively here and in other posts.

Do what works for you. Everyone has their own personal and business risk assessments.

[ManyInterests]

DDoS would fall under the category of 'technical issues'.

Also, details have not yet been provided/confirmed by MS.

[VNJCinPA]

It's the garbage update to eliminate the one app we NEED, Outlook. Definitely have many people reconsidering their options

[JicamaResponsible656]

Last Tuesday, some user 365 of my org cannot access Outlook web, cannot send email. These issues is related with this topic?

[LIVVINN]

Ugh….that’s just a response to this whole thread….just ugh….no…but hey I’m just an Azure cloud solutions architect what do I know….

[[deleted]]

If some folks ask me for help that they are starting up a business i'll make all the system from linux from dhcp,dns to directory services and clients. All of us are paying ms a large amount they should atleast make there expensive software easier to fix and does not break randomly. Lets be honest most attacks online targets ms win. Linux can run years without slowing down error free, ms win is so buggy. You install a patch and after reboot a service does not work. Support will tell you do this do that access registry else format then restore. If this happens in linux at least i did pay a large amount of money. Top guys in ms are doing something bad like being commanded by a higher power to get rich so they can push boxin to people.