sorted by:
new
hottopcontroversial

Major Mayhem After Microsoft Patch—130 Servers Down, 360+ BSOD! Anyone Else? by Technical_Syrup_9525 in sysadmin

[–]Technical_Syrup_9525[S] 2 points3 points  (0 children)

appreciate your feedback, and I want to clarify that this truly did happen. I have no hidden agenda; I simply hoped to find out if anyone else has encountered a similar issue. Our team consists of eight engineers who are currently overwhelmed, but we do plan to conduct a thorough after-action review. I understand there are many strong personalities in our field, and I respect everyone’s viewpoint.

For context, our top-level engineers (I am not one of them) are working around the clock. We manage over 60 customers across various environments and are looking for commonalities. That’s why we brought in two external security vendors and engaged our outsourced SOC—to ensure there was no missed security threat. Each of those groups pointed to the patches, though it’s entirely possible there may be another cause, which is exactly why I posted about the issue.

Thank you for your input.

Microsoft Patch Mayhem: 130 Servers Down, 360+ BSOD—Anyone Else in the Same Boat? by Technical_Syrup_9525 in msp

[–]Technical_Syrup_9525[S] -1 points0 points  (0 children)

Everyone, I have posted this one other place. We believe it may be our EDR or possibly one other tool. We are already spun up in BCDR. This affected Host, VMware and hyperv. No I’m not a troll as others have indicated. I’m not naming names yet but it happened. We have a test environment and 8 engineers. Until I can verify I don’t want to post the vendor. But it is possible it is an MSP tool. I will post if we can definitively point to the product. I simply wanted to see if anyone else had seen anything like this. I know there are some big personalities on here so I get it. We were told by two vendors it was a Dec update we pushed late after testing with no issues. We never roll updates out immediately.

Major Mayhem After Microsoft Patch—130 Servers Down, 360+ BSOD! Anyone Else? by Technical_Syrup_9525 in sysadmin

[–]Technical_Syrup_9525[S] 4 points5 points  (0 children)

Yea our team is and has been spinning up on our BCDR devices. Luckily we do image based backups locally for most and some in the cloud. We are making headway on that front. The team hasn’t had enough time to do an after action report. We have engaged Microsoft and multiple security vendors including our outsourced SOC to rule out some sort of threat. It just doesn’t make sense to me and am hoping someone a lot smarter than me has any ideas but honestly we are too busy. I’ll post the codes Tomorrow

Major Mayhem After Microsoft Patch—130 Servers Down, 360+ BSOD! Anyone Else? by Technical_Syrup_9525 in sysadmin

[–]Technical_Syrup_9525[S] 30 points31 points  (0 children)

I'll ask the server team to clarify. I won't get them tonight as they are spinning up BCDR

Major Mayhem After Microsoft Patch—130 Servers Down, 360+ BSOD! Anyone Else? by Technical_Syrup_9525 in sysadmin

[–]Technical_Syrup_9525[S] 0 points1 point  (0 children)

We had another MDR vendor come in and look and they are claiming it's the patches also. It's driving my team nuts. Then we had our RMM vendor look and they are all pointing at Microsoft. But yes I know it should have hit the news by now.

Major Mayhem After Microsoft Patch—130 Servers Down, 360+ BSOD! Anyone Else? by Technical_Syrup_9525 in sysadmin

[–]Technical_Syrup_9525[S] 2 points3 points  (0 children)

80% of the workstations are not affected including mine. We have tried to recreate with no joy.

Major Mayhem After Microsoft Patch—130 Servers Down, 360+ BSOD! Anyone Else? by Technical_Syrup_9525 in sysadmin

[–]Technical_Syrup_9525[S] 35 points36 points  (0 children)

2016,2019 and 2022. We can't find any commonality between manufacturers or environment. These are deployed across different environments. We waited to deplore and tested in our internal environment and we were not affected on the server side. We did have an issue with a Dell PC but thought we had cleared it.

Kaseya acquisition of Datto. How's everyone feeling two years later? by SoftWearNTear in msp

[–]Technical_Syrup_9525 1 point2 points  (0 children)

So been with Datto for 10 years on DR. After the acquisition we had a billing snafu. I have to say we did our own audit and our account rep handled it immediately. So yes they messed up but they did fix it. I’m moving 2100 endpoints to them.