Summary: Quorum is one of the most important aspect of a Database Availability Group (DAG). It helps determine if the DAG should remain online or go offline. To prevent DAG from going offline and cause outage, it's important to ensure quorum is maintained. If you are receiving the "Database Availability Group must have Quorum” error, you must fix it immediately to ensure DAGs' automatic recovery and site-resilience features.
Contents
A Database Availability Group (DAG) works great for failover and high availability for your Exchange Server and services. Although this is the best solution around for Exchange Server business continuity and disaster recovery, you may encounter some unforeseen issues that might hinder the performance, quality, and data integrity of the Exchange Server infrastructure.
One of the worst scenarios that an Exchange Administrator might encounter is when a mailbox database is dismounted. When you try to mount the database, you may encounter the following error:
Failed to mount database “DB01“. Error: An Active Manager operation failed. Error: An Active Manager operation encountered an error. To perform this operation, the server must be a member of a database availability group, and the database availability group must have quorum. Error: Unable to read from the cluster database.
This may occur if the server has recently been removed from the database availability group.
[Server: EX2019.mail.lan] + CategoryInfo : InvalidOperation: (DB01:ADObjectId) [Mount-Database], InvalidOperationException + FullyQualifiedErrorId : [Server=EX2019,RequestId=c51a831c-33e3-4a75-867f-51433b925ee2,TimeStamp=1/1/2020 4:13:12 AM] [FailureCategory=Cmdlet-InvalidOperationException] 7E29F70A,Microsoft.Exchange.Management.SystemConfigurationTasks.MountDatabase + PSComputerName : EX2019.mail.lan
The above error could happen due to various reasons. Some of the reasons are:
- An update gone wrong
- Human error
- Hardware failure
- Power loss
- Malware and ransomware attack
- Corruption or storage issues
- Failure on the server?s storage unit
Fix ?Database Availability Group must have Quorum? Error
The first thing to check is that you have the voting majority in your cluster. It means that you have half of your servers, plus one online and all participating servers are healthy and running. Also, you need to check that the storage of each node, along with the services of Exchange and the Cluster Service, are running and no errors occurred. If you have suffered a hardware issue that has powered down the server, this might be an issue as the DAG cannot function with one server, out of three. Like in other clusters (for example, SQL), if the voting majority is not reached, the cluster will shutdown as a safety precaution.
If all troubleshooting has been done and there is nothing you can do to restore the services back to normality, the only option is to remove the database copies so that the server would be in single database mode.
This can be done by running the Remove-DatabaseAvailabiltyGroupServer command from the Exchange Management Shell. This command will clear off any DAG entries from the Active Directory.
Note: There is no need to back up everything beforehand as this will remove just the configuration.
Remove-DatabaseAvailabilityGroupServer -ConfigurationOnly -MailboxServer SRV-MBX-01 -Identity DAG001
This process needs to be performed on every node. So, if your Database Availability Group (DAG) is made up of two servers, you need to perform the above on both servers.
Once this operation is done, the Database Availability Group (DAG) configuration will be removed from the Active Directory.
The next step is to evict the nodes from the Windows Cluster. But you cannot use the Cluster Manager for this operation. It must be done by using the PowerShell. The operation is quite simple as you just need to use the Get-ClusterNode and the Remove-ClusterNode using the force parameter (as per the below example).
Get-ClusterNode <server name> | Remove-ClusterNode -Force
If the Exchange Server setup for the Database Availability Group (DAG) is made up of two or more server, you would need to perform this for each node in your cluster.
After this, any configuration of the Database Availability Group is removed from the Active Directory Schema and the setup should be converted as stand-alone server. This means that after a reboot of the servers, the database should mount as it is cleared from any Database Availability Group (DAG) membership.
What to do if the Databases still not Mount?
Now, if we?re saying that the databases should be mounted with no issues, it implies that no damage has occurred to the databases. If the databases still not mount, you might consider checking the databases by using the native tools that come with Exchange Server, such as the ESEUtil.
ESEUtil is a native tool which identifies issues and tries to repair corrupted databases. It must be run from within a command line interface.
To identify if the database has any damage, you need to run the ESEUtil with the /mh parameter and check the state of database. A healthy database would have Clean Shutdown state, while a damaged database will show Dirty Shutdown.
EseUtil /mh <exchange database file>
ESEUtil offers two database recovery options ? Soft Recovery and Hard Recovery. The Soft Recovery process, as the name implies, runs a soft/quick recovery of the database. If the damage is light, the Soft Recovery will try to fix the issue.
If this fails and the database state remains in Dirty Shutdown, this means that the corruption is severe. In such a case, the only solution is to restore the databases from backup. But this will result in data loss as you?re only able to restore data from the last backup. The other option is to perform the Hard Recovery. This also results in data loss as Hard Recovery process deletes any data which is deemed corrupted. On the other hand, Microsoft will not assist you if the database is found hard recovered. In addition, there is no guarantee that the database will mount afterwards.
As a healthy alternative, you can use a third-party application, like Stellar Repair for Exchange that can easily open any Exchange Server version database in any state and you will be able to read and browse the EDB file. You will be able to export the data to PST and other formats. The Exchange Server recovery software also has a feature to export directly to a live Exchange Server database or Office 365 tenant.
