Response to CrowdStrike Falcon Sensor Agent Issue Affecting Microsoft Devices

Response to CrowdStrike Falcon Sensor Agent Issue Affecting Microsoft Devices

As of 0409 UTC, a critical issue with CrowdStrike Falcon Sensor agents on Windows devices has caused significant global outages. This was not a security incident or cyberattack, and DirectDefense’s infrastructure was not affected by this outage.

The root cause has been identified as an automatic content deployment applied to Windows hosts, which has resulted in blue screen errors and system instability. Mac and Linux systems remain unaffected. CrowdStrike reports that the issue has been identified, isolated and a fix has been deployed.

As part of our commitment to transparent communication, we have diligently relayed updates from both CrowdStrike and Azure regarding this issue to our clients, assisting them with rollback and remediation steps as needed. 

Support details: 

• Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor. 
• Windows hosts which have not been impacted do not require any action as the problematic channel file has been reverted. 
• Windows hosts which are brought online after 0527 UTC will also not be impacted 
• Hosts running Windows7/2008 R2 are not impacted. 
• This issue is not impacting Mac- or Linux-based hosts 
• Channel file “C-00000291*.sys” with timestamp of 0527 UTC or later is the reverted (good) version.
• Channel file “C-00000291*.sys” with timestamp of 0409 UTC is the problematic version. 

Workaround steps for individual hosts: 

Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then: 

• Boot Windows into Safe Mode or the Windows Recovery Environment 
• Note: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation. 
• Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory 
• Note: On WinRE/WinPE, navigate to the Windows\System32\drivers\CrowdStrike directory of the OS volume 
• Locate the file matching “C-00000291*.sys”, and delete it. 
• Boot the host normally. 

Note: Bitlocker-encrypted hosts may require a recovery key. 

Workaround steps for public cloud or similar environment including virtual: 

Option 1: 

1. Detach the operating system disk volume from the impacted virtual server
2. Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
3. Attach/mount the volume to to a new virtual server
4. Navigate to the %WINDIR%\\System32\drivers\CrowdStrike directory 
5. Locate the file matching “C-00000291*.sys”, and delete it. 
6. Detach the volume from the new virtual server 
7. Reattach the fixed volume to the impacted virtual server 

Workaround Steps for Azure via serial: 

Option 2:  Roll back to a snapshot before 0409 UTC. 

1. Login to Azure console –> Go to Virtual Machines –> Select the VM 
2. Upper left on console –> Click : “Connect” –> Click –> Connect –> Click “More ways to Connect”  –> Click : “Serial Console” 
3. Step 3 : Once SAC has loaded, type in ‘cmd’ and press enter. 
4. type in ‘cmd’ command 
5. type in : ch -si 1 
6. Press any key (space bar). Enter Administrator credentials
7. Type the following: 
   • bcdedit /set {current} safeboot minimal
   • bcdedit /set {current} safeboot network 
8. Restart VM 
9. Optional: How to confirm the boot state? Run command: 
   • wmic COMPUTERSYSTEM GET BootupState 

Detailed statements from both CrowdStrike and Microsoft can be found on their respective company support websites. 

• CrowdStrike Statement 
• CrowdStrike Tech Alert 
• Azure Status 

Additional CrowdStrike Resources

• Tech Alert | Windows crashes related to Falcon Sensor
• BitLocker recovery in Microsoft environments using IBM BigFix 
• BitLocker recovery in Microsoft environments using ManageEngine Desktop Central
• BitLocker recovery in Microsoft environments using Ivanti Endpoint Manager
• BitLocker recovery in Microsoft environments using Active Directory and GPOs
• BitLocker recovery in Microsoft environments using SCCM
• BitLocker recovery in Microsoft Azure

DirectDefense clients may continue to work with their SOC team for any additional support. CrowdStrike customers who are not DirectDefense MSSP clients may refer to the CrowdStrike support portal for the latest updates.

   By: Charly Bun   07.19.24