Having frustrated users is one thing but having the database unavailable due to the fact that it doesn’t mount is a disaster. After researching and investigating the issues the problem was due to database corruption which usually can be caused by hardware or software failure. In my situation it was an unexpected power off of the virtual machine.
I have tried to move the database and log file to another location. After the copy was complete I have executed the EseUtil command using the /p parameter.
As soon as I run the command after some time I get the “Operation terminated with error -1605 (JET_errKeyDuplicate, Illegal duplicate key” error. At this stage, try to re-run the EseUtil again but this time with the /mh parameter don’t get any issues and the database status is showing as clean shutdown.
At this stage unfortunately I was in a sticky situation as the use users were frustrated without being able to send or receive emails. The EseUtil is showing a clean shutdown but the database is clearly corrupted. The only solution is to execute a hard recovery of the database by again using the EseUtil with the /P parameter but this will repair the database by discarding the databases pages that cannot be repaired. It’s nothing bad but this is how it works. It will try to repair the database by removing the bad parts from it, this means data loss. After running this which doesn’t mean that your database might mount.
TIP: This occurs when the Eseutil generates inaccurate B-trees. You should always backup the database in such scenario to avoid any further damage. Additionally, you can use an Exchange recovery software, such as Stellar Repair for Exchange to repair the database that fails to mount even though Eseutil command displays the database in clean shutdown state.
After this process is complete, we might need to defragment the database and this can be done by using the IsInteg tool. If you are running Microsoft Exchange 2013 Server with SP1 or later, you might consider using the New-MailboxRepairRequest PowerShell cmdlet which can be executed on the database. It’s important that if you execute the command you will not be able to stop it without damaging the database. The only way to stop it is by unmounting the database. One good note on this is that once you are running the New-MailboxRepairRequest the mailboxes will remain accessible of course note that this process will take a considerate amount of time depending on the size of the database.
If this doesn’t help and you are stuck with a database that cannot be mounted, the only solution would be to restore from the backup. The only problem with this that you would lose all the data from the last backup. I know it’s a very bad solution, but unfortunately if you are against a wall that’s the only solution. So what to do with the lost data of the day? The only way to solve this is to go on each and every user and extract the data from the OST file. This cannot be done centralized and it must be done on every computer. You would need to export to PST manually the lost emails. You will not recover everything as the emails could have been moved to folders and deleted which will cause an issue with the recovery. The other problem is time. This process will take you a lot of time to complete and an administrative nightmare.
The other thing is that if you have remote users, or clients connected with Exchange not using Exchange Cached Mode, they will not be able to recover anything. Since it’s a direct connection to the Exchange, nothing is cached on the local machine or device. This can be a real grave situation as the lost emails are not recoverable. Depending on the number of mailboxes, database size and performance of the server; again this will take a considerate amount of time and when you have a big number of users who cannot access or send emails and cannot receive emails is a huge impact on the business and possibly your position. No time can be wasted on this will mean big money for the boss. Ok, you can make a quick SMTP server just to collect the incoming emails so that these are not lost but then again, the users will not be able to access them.
Not to waste time the ideal solution is to create a new database and re-create the mailboxes for the users. The problem would be that all the data, corrupted or not is still in the EDB file. There is no native way to export these so the only solution is to try one of the third party Exchange Recovery Tool to export all the mailboxes and import them into the new mailbox database.
There are a lot of application than can do this job but Stellar Repair for Exchange is the best contender for the job. As said you can create a new mailbox database so that the users can immediately start receiving and sending new emails, create all mailboxes and let the application do the rest. Once opened just point it to the corrupted EDB file and after a brief scan (depending on the size) you will have a tree of all the mailboxes in the database. You can always export to PST and import them into the new database, but will Stellar Repair for Exchange you can re-import the mailboxes directly into the newly created database. The process is fast and reliable. Of course you can export from the EDB file to PST with a huge number search criteria and you can also export directly to Office 365.