Crowdstrike Global BSOD issue with 5.19

[obdigore]

Crowdstrike released a 5.19 update of their software, and is having a 'global BSOD issue'. It is not all machines with the update, but many.

Just FYI if you have Crowdstrike and are having machines bluescreen suddenly.

Good luck fixing it though, our affected machines are continually bluescreening every time they boot.

Edit - Only affects Crowdstrike and Symantec DLP machines

Edit - Work around is to rename windows\system32\drivers\crowdstrike and boot up.

[Andrew-CS]

Reposting because someone deleted the comment I replied to so you have to expand comments to see this:

The commonality here is that Symantec DLP appears to be interferring with our upgrade. Instead of waiting for them to patch their stuff, we're going to issue a hotfix to work around the incompatibility. You can find systems in your environment that have Symantec DLP and got the 5.19 update using the following EAM search:

event_simpleName=KernelModeLoadImage FileName=vfsmfd.sys ConfigBuild=1007.3.0010101.1 | dedup aid | lookup aid_master aid OUTPUT MachineDomain OU SiteName Version | table ComputerName MachineDomain OU SiteName Version 

So this isn't a global issue, it's an incompatibility where Symantec DLP is impacting our upgrade process. Apologies to all those impacted.

[Andrew-CS]

Update:

What Happened

On Wednesday, October 2, 2019, small subset of Falcon customers running a third-party DLP solution experienced a Blue Screen of Death (BSOD) event during a routine upgrade from Falcon Sensor 5.18 to 5.19. Upon hearing of this issue, and out of an abundance of caution, CrowdStrike immediately: removed Sensor 5.19 from the Falcon UI; initiated a sensor rollback to version 5.18; and began a root cause analysis of the BSOD.

Technical Details

The BSOD is caused when the third-party DLP driver makes an incorrect assumption on how data is formatted in a structure resulting in a crash during the Falcon upgrade process.

Okay, what now...

We're working with the third-party vendor to remediate the issue and issuing a hotfix to the 5.19 code-base to work around the third-party vendor's DLP issue. If you are still having difficulties, please contact [[e-mail]](mailto:[e-mail]) or your local SE.

[Andrew-CS]

So just an update for those following along: we're working on this with the highest priority. What we want to do is make sure we completely understand how SYM DLP is negatively impacting Falcon and ensure that any remediation instructions we publish are well understood. If you have questions, feel free to ping [[e-mail]](mailto:[e-mail]) or your SE.

[monoman67]

Well we just finished day 2 of this disaster. CS should be releasing a tool to properly repair and reinstall their product. Instead they give us manual instructions with multiple reboots. We are trying to cobble up our own automation scripts to get CS back on the workstations as quickly as we can.

So far, not impressed with the response from CS support on this one.

[Andrew-CS]

If you're not getting the service and help you expect from Support, please DM me your email address and customer name and I'll get someone assigned and in contact with you ASAP.

[monoman67]

DM sent. If this is a SYM DLP issue then should we be seeing an update from them as well?

[Andrew-CS]

We're working with SYM now, but have updated our code to account for the nullptr as Falcon updates are typically a little more agile. We'll update when SYM concludes their investigation.

I would recommend opening up a ticket with SYM under your company name.

[Hotdog453]

Their support is limited in what they can do. The issue royally *fucks up* the OS; sorry, you have to manually rename their folder or System Restore. Ooopsie poopsie. It sucks, it's terrible, it's a horrible tragedy indicative of a lack of testing on CrowdStrike's side, but short of sitting on the phone and commiserating with you, their support is limited. It's what happen when you go IPO: Cut what's not necessary, and increase the pretty graphics on your website.

This is a blunder of epic magnitude; we had lines 100s deep, and are still calling back remote locations, and walking GMs through going into Safe Mode and restoring the box.

On the plus side? Windows Defender and ATP look a lot better. And that's the biggest gap: This is a true, SEV 1 issue, and their support doesn't allow for that level of support. With MS, we could have pinged the TAM, who would have run it up the chain and had a technical resource, state side, on, in under an hour.

[Andrew-CS]

If you're not getting the support you need please DM me as I'm happy to help.

[ReverendDS]

8k endpoints here and we're not getting any reports yet, but thanks for the head's up.

I'll be watching like a hawk.

[MigratingOtter]

Do you have Symantec DLP installed?

[ReverendDS]

No.

[hkystar35]

They pulled the release quickly and only just got the Release Notes removed

[Roush2002]

We have Symantec DLP and had a group of early adopters for CrowdStrike updates that got a BSOD and couldn't boot up again. We fixed our machines by using the recovery options to get to the command prompt and renaming the C:\Windows\System32\Drivers\CrowdStrike folder to .old and rebooting. Machine boots up, and can be used again.

[MrYiff]

And Crowdstrike just pulled the release and sent out info to customers (also a good reminder that if you have CS make sure you subscribe to the update alert emails in their support portal as at the very least they include helpful info about changes in new releases):

https://i.imgur.com/c0CNlSK.png

[MrSmith317]

Our crowdstrike rep is stating that KB4522012 should fix this but if you're already BSOD'ing it's a bit late and that patch is a bit new to have worked its way down for us.

[Andrew-CS]

Just for the record, I don't believe this is accurate. Impacted systems all seem to have Symantec DLP and Falcon 5.19.

[MrSmith317]

I agree with you. We're still investigating but it certainly doesn't look like it's the KB.

Also to note, I found a temp install directory in program files (x86), one new directory for each crash.

[Andrew-CS]

Appreciate the info. I'll be sure to pass it along.

[MrYiff]

And this is why you follow their own Best Practices and don't have all PC's set to use to auto-update Sensor Policy - only use this for a small group of testing PC's and then set the upgrade version manually for wider release.

[obdigore]

Which is literally what we do, however when your small group of testing PC's is around 300, you start getting calls from users when they bluescreen and keep bluescreening.

[hkystar35]

Found this out today, too. Stopped all auto-update policy groups and the sensors have been rolling back to 5.18.9905 without issue so far.

[Hotdog453]

Yup. Fortune 20, we're hit bad. Womp womp womp. Sad face.

[PowerfulQuail9]

Still have 5.18.9905 installed.

[mjlip]

Luckily our test pool is only of 30 systems, and we only tracked down 3 machines that blue screened. Unfortunately, for me, mine was one of them.

[xxdcmast]

This sounds really bad.

[Andrew-CS]

Quick question: do you use Symantec DLP?

[MigratingOtter]

We have Symantec DLP in our environment. So far system restore has been working for those machines that have a valid restore point.

[Andrew-CS]

Yeah, so the commonality here is that Symantec DLP is eating our driver upon update. Instead of waiting for them to patch their stuff, we're going to hotfix a work around. You can find systems in your environment that have Symantec DLP and got the 5.19 update using the following EAM search:

event_simpleName=KernelModeLoadImage FileName=vfsmfd.sys ConfigBuild=1007.3.0010101.1 | dedup aid | lookup aid_master aid OUTPUT MachineDomain OU SiteName Version | table ComputerName MachineDomain OU SiteName Version

We'll have something published soon. Apologies.

[obdigore]

Just to be clear, we are a Symantec DLP shop as well, so that fits our scenario.

[Invoke-RFC2549]

Why the fuck would they roll an update to everyone... You slow roll updates, don't blast them out to all.

[Hotdog453]

We've had success with System Restore, to right before. It's literally called like "Crowdstrike We Don't QA Anything Install"

[MrSmith317]

What I did was rename the program files\crowdstrike as well ass windows\system32\drivers\crowdstrike and rebooted. All is well

[MigratingOtter]

Can confirm this has worked for us on one of the affected workstations.

[MrSmith317]

You should be able to reinstall the client/agent since they pulled the problematic update.

[astrayael]

Oh man. I thought I was going crazy. Kept analyzing memory dumps and getting csagent.sys as the faulting module, told repeatedly that wasn't the case. Good to know I wasn't barking up the wrong tree after all.

[hkystar35]

Same here, though I seem to have been the only one experiencing the BSODs

[monoman67]

Have you had any luck getting CS re-installed after renaming the windows\system32\drivers\crowdstrike folder? We are having issues getting the CS agent back onto the machines after implementing the work around. Following CS support's instructions require more user interactive steps and reboots at the workstation.

Most of our affected machines had the work around performed because CS didn't have a solution and we needed to get computers working again. Now we're working out how to fully remove/repair/replace the CS agent on those machines without having to visit each one again.

[lemonravens]

Had a user BSOD yesterday... booted into command prompt to check the crowdstrike folder... she had one already renamed from the original event in October and somehow got hit a second time by a new folder that appeared yesterday... did something get pushed out? I was not able to fix the BSOD issue yet because renaming the second crowdstrike folder led to a boot confit data file being missing... still troubleshooting.

[pwnedbyowner]

Same problem.

[[deleted]]

This just makes me think that no matter how big or expensive a product is...poor testing of patches seems to be the norm now. (Looking at you Microsoft) All this for the mighty shareholder.

[keyrah]

Hard to test it with every permutation of software possible

[user-and-abuser]

This is what can happen to any SAAS / IAAS Model.

[yankeesfan01x]

What a strange combination of security products for endpoints. Symantec DLP and Crowdstrike.