Summary: The Year 2022 started with a new bug in On-premises Microsoft Exchange's email delivery system, which interrupted New Year's celebrations of many administrators across the globe. The bug dubbed Y2K22 is causing messages to be stuck in the transport queues of On-premises Exchange Server 2016 and 2019. In this blog, we have addressed the Y2K22 bug and discussed a solution to resolve the problem.
Microsoft Exchange On-premises Servers stopped delivering messages at the beginning of January 1, 2023, midnight (UTC) due to an issue in the FIP-FS antimalware scan engine with the following error message,
FIP-FS Scan Engine failed to load ? "Error code: 0x80004005. Error Description: Can't Convert "2201010001" to long (2023/01/01 00:00 UTC)" or "Error Code: 0x80004005 ? Can't convert "2201010001" to long."
This is a non-security issue, which isn’t related to malware scanning. Instead, the problem is caused by the date check failure with the new year’s change, causing the malware scan engine to crash, resulting in messages getting stuck in the transport queues of On-premises Exchange Servers.
Exchange Server download daily updates to ensure malware scan engine can detect newer threats and malware. After the download, the server checks the version using the date. However, with the year 2023, the date check fails with even ID 5300 and 1106 (FIPFS).
Log Name: Application Source: FIPFS Logged: 1/1/2023 1:03:42 AM Event ID: 5300 Level: Error Computer: server1.contoso.com Description: The FIP-FS "Microsoft" Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long.
Log Name: Application Source: FIPFS Logged: 1/1/2023 11:47:16 AM Event ID: 1106 Level: Error Computer: server1.contoso.com Description: The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error.
The date check failure is caused by Microsoft using a ‘long’ Int32 variable for storing the date value, which has a maximum value of 2,147,483,647. However, the new dates in 2023 have a minimum value of 2,201,010,001, exceeding the maximum value that the long Int32 variable can store.
NOTE: Edge Transport Servers are unaffected by the Y2K22 bug.
Solutions to Resolve Y2K22 Exchange Server Bug
Microsoft has acknowledged the issue and will soon release an update with a new variable that can hold the date to fix this issue.
For now, admins can apply the following solutions on the affected on-premises Exchange Server to temporarily fix the problem and resume email delivery.
Solution 1: Disable FIP-FS Scanning Engine
This unofficial fix shared by admins can help you fix the Y2K22 bug and enable on-premises Exchange Servers to start delivering emails with a caveat. Once you disable the FIP-FS Scanning engine, Microsoft’s scan engine will not scan or filter the emails or messages. As a result, the spam and malicious emails will bypass the scan engine and be delivered to users.
To disable the FIP-FS scanning engine, open PowerShell as administrator and execute the following command,
Set-MalwareFilteringServer -Identity -BypassFiltering $true
Then restart the MSExchangeTransport service by executing the following command in the PowerShell.
Restart-Service MSExchangeTransport
Once the commands are executed successfully, the email delivery will start, and messages will no longer be stuck in the transport queue.
Solution 2: Run Reset-ScanEngineVersion.ps1 PowerShell Script
Microsoft has released a ResetScanEngineVersion.ps1 PowerShell script that you can run on the affected On-premises Exchange Servers to fix the Y2K22 bug temporarily and resume email delivery.
The steps are as follow,
- Download the Reset-ScanEngineVersion.ps1 PowerShell script on the affected on-premises Exchange Server.
- Open Exchange Management Shell (EMS) as administrator and execute the following command to change the execution policy
- Then navigate to the download folder where ResetScanEngineVersion.ps1 is located using the ‘cd‘ command. For instance,
cd C:\Users\administrator\Downloads\
- Then execute the Reset-ScanEngineVersion.ps1 PowerShell script by running following command,
.\Reset-ScanEngineVersion.ps1
The script stops the Exchange services, removes the Microsoft engine folder, empties the metadata folder, and restarts the services. After this, it updates to the latest engine.
Solution 3: Manually Update the Microsoft Engine
Instead of using the ResetScanEngineVersion.ps1 PowerShell script, users may also manually perform the steps to resolve the Y2K22 bug and restore the services. The steps are as follow,
Step 1: Stop Microsoft Filtering Management Service. This will also stop the Microsoft Exchange Transport Service. Next, open the Task Manager and check updateservice.exe is not running.
Step 2: Delete the %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft folder.
Step 3: Delete all the files from the following location %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata
Step 4: Start the Microsoft Filtering Management Service and Microsoft Exchange Transport Service
Step 5: Navigate to %ProgramFiles%\Microsoft\Exchange Server\V15\Scripts and execute the Update-MalwareFilteringServer.ps1 PowerShell script using the following command,
Update-MalwareFilteringServer.ps1 <server FQDN>
Step 6: Open Exchange Management Shell and run the following commands to update the latest engine.
Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell
Get-EngineUpdateInformation
In the output, check the UpdateVersion information. It should be ‘2112330001.’
Then use the Get-Queue command to check that the queues are draining.
NOTE: Depending on the size of your organization and the number of messages in the queues, the queue draining may take a while to finish.
To Wrap Up
Microsoft has postponed the December 2021 Cumulative Update but will soon release a patch to fix this particular problem with On-premises Exchange Server 2016 and Exchange Server 2019. For now, you can apply the solutions discussed above on affected on-premises Exchange Servers to resolve the Y2K22 bug and resume the email services. After applying the solutions, check that the mail flow is working and there are no FIP-PS errors or issues in the Application event log.
However, this problem does not exist on Exchange Servers that are not connected to the internet or download the antimalware updates.
