Y2K22 Bug in Microsoft Exchange Breaks Email Delivery

Microsoft Exchange On-premises Servers stopped delivering messages at the beginning of January 1, 2022, midnight (UTC) due to an issue in the FIP-FS antimalware scan engine with the following error message,

FIP-FS Scan Engine failed to load – "Error code: 0x80004005. Error Description: Can't Convert "2201010001" to long (2022/01/01 00:00 UTC)" or "Error Code: 0x80004005 – Can't convert "2201010001" to long."

This is a non-security issue, which isn’t related to malware scanning. Instead, the problem is caused by the date check failure with the new year’s change, causing the malware scan engine to crash, resulting in messages getting stuck in the transport queues of On-premises Exchange Servers.

Exchange Server download daily updates to ensure malware scan engine can detect newer threats and malware. After the download, the server checks the version using the date. However, with the year 2022, the date check fails with even ID 5300 and 1106 (FIPFS).

Log Name: Application
Source: FIPFS
Logged: 1/1/2022 1:03:42 AM
Event ID: 5300
Level: Error
Computer: server1.contoso.com
Description: The FIP-FS "Microsoft" Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long.

Log Name: Application
Source: FIPFS
Logged: 1/1/2022 11:47:16 AM
Event ID: 1106
Level: Error
Computer: server1.contoso.com
Description: The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error.

The date check failure is caused by Microsoft using a ‘long’ Int32 variable for storing the date value, which has a maximum value of 2,147,483,647. However, the new dates in 2022 have a minimum value of 2,201,010,001, exceeding the maximum value that the long Int32 variable can store.

NOTE: Edge Transport Servers are unaffected by the Y2K22 bug.

Solutions to Resolve Y2K22 Exchange Server Bug

Microsoft has acknowledged the issue and will soon release an update with a new variable that can hold the date to fix this issue.

For now, admins can apply the following solutions on the affected on-premises Exchange Server to temporarily fix the problem and resume email delivery.

Solution 1: Disable FIP-FS Scanning Engine

This unofficial fix shared by admins can help you fix the Y2K22 bug and enable on-premises Exchange Servers to start delivering emails with a caveat. Once you disable the FIP-FS Scanning engine, Microsoft’s scan engine will not scan or filter the emails or messages. As a result, the spam and malicious emails will bypass the scan engine and be delivered to users.

To disable the FIP-FS scanning engine, open PowerShell as administrator and execute the following command,

Set-MalwareFilteringServer -Identity  -BypassFiltering $true

Then restart the MSExchangeTransport service by executing the following command in the PowerShell.

Restart-Service MSExchangeTransport

Once the commands are executed successfully, the email delivery will start, and messages will no longer be stuck in the transport queue.

Solution 2: Run Reset-ScanEngineVersion.ps1 PowerShell Script

Microsoft has released a ResetScanEngineVersion.ps1 PowerShell script that you can run on the affected On-premises Exchange Servers to fix the Y2K22 bug temporarily and resume email delivery.

The steps are as follow,

• Download the Reset-ScanEngineVersion.ps1 PowerShell script on the affected on-premises Exchange Server.
• Open Exchange Management Shell (EMS) as administrator and execute the following command to change the execution policy
• Then navigate to the download folder where ResetScanEngineVersion.ps1 is located using the ‘cd‘ command. For instance,

cd C:\Users\administrator\Downloads\

• Then execute the Reset-ScanEngineVersion.ps1 PowerShell script by running following command,

.\Reset-ScanEngineVersion.ps1

The script stops the Exchange services, removes the Microsoft engine folder, empties the metadata folder, and restarts the services. After this, it updates to the latest engine.

Solution 3: Manually Update the Microsoft Engine

Instead of using the ResetScanEngineVersion.ps1 PowerShell script, users may also manually perform the steps to resolve the Y2K22 bug and restore the services. The steps are as follow,

Step 1: Stop Microsoft Filtering Management Service. This will also stop the Microsoft Exchange Transport Service. Next, open the Task Manager and check updateservice.exe is not running.

Step 2: Delete the %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft folder.

Step 3: Delete all the files from the following location %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata

Step 4: Start the Microsoft Filtering Management Service and Microsoft Exchange Transport Service

Step 5: Navigate to %ProgramFiles%\Microsoft\Exchange Server\V15\Scripts and execute the Update-MalwareFilteringServer.ps1 PowerShell script using the following command,

Update-MalwareFilteringServer.ps1 <server FQDN>

Step 6: Open Exchange Management Shell and run the following commands to update the latest engine.

Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell

Get-EngineUpdateInformation

In the output, check the UpdateVersion information. It should be ‘2112330001.’

Then use the Get-Queue command to check that the queues are draining.

NOTE: Depending on the size of your organization and the number of messages in the queues, the queue draining may take a while to finish.

To Wrap Up

Microsoft has postponed the December 2021 Cumulative Update but will soon release a patch to fix this particular problem with On-premises Exchange Server 2016 and Exchange Server 2019. For now, you can apply the solutions discussed above on affected on-premises Exchange Servers to resolve the Y2K22 bug and resume the email services. After applying the solutions, check that the mail flow is working and there are no FIP-PS errors or issues in the Application event log.

However, this problem does not exist on Exchange Servers that are not connected to the internet or download the antimalware updates.