Azure Hybrid & Cloud Migration Blog

Author: Zahir Hussain Shah | MVP Exchange, CISSP

Managing Virtualized Domain Controller | Recovering Domain Controller from UNS Roll-back | Best Practices for Domain Controller VMs

Recently I saw a situation, where a Domain Controller on Windows Server 2008 based Virtual Machine (VM) running on Windows Server 2008 SP2 Hyper-V, was tried to move from one Hyper-V Server to another Hyper-V Windows Server 2008 R2 SP1, initially the VM was shut down normally, and then copy the VHD to another Hyper-V Server, and then created new VM with attaching the copied VHD, and as expected the VM didnt had IP Address, so after giving the IP Address, machine was pinging, but at the moment of checking the Active Directory Replication, I saw below errors:

Problem:

From Active Directory Sites and Services snap-in, if we try to replicate now for all the available connections, we receive errors:

o The Source / Destination Domain Controller is rejecting replication from the Destination / Source.

Upon running, Repadmin /syncall command from the CMD, you present with below error:

o SyncAll reported the following errors:

Error issuing replication: 8452 (0x2104):

The naming context is in the process of being removed or is not replicated from the specified server.

From: 730663f3-e425-4041-b969-a34b5b241af0._msdcs.domain.net

To : 3920866b-fecd-4a9d-8c29-97a20e307517._msdcs.domain.net

Cause:

When the VM was moved from the running Hyper-V Server to the newly configured Hyper-V Server, where the VM configuration file got created new, and since the VM was down for bit long time, so when it brought up, and communicated with the other Domain Controller, it found that it has lower USN number, which indicated as old Active Directory database, thus as part of built-in Windows Server 2008 and Windows Server 2003 SP1 security mechanism for Active Directory, the domain controller has marked himself as NON-Writable Domain Controller, and started rejecting for replication.

Symptoms

You can verify your problem with the below mentioned symptoms of the problem, so if you have the exact same problem, then you may follow the below steps to recovery your Domain Controller:

AD DS pauses the Net Logon service, which prevents user accounts and computer accounts from changing account passwords. This action prevents the loss of such changes if they occur after an improper restore.

AD DS disables inbound and outbound Active Directory replication.

AD DS generates Event ID 2095 in the Directory Service event log to indicate the condition.

Solution:

Before, we move forward with the resolution, I would like to address here few of the Best Practices for virtualizing Domain Controller, these best practices are not Hyper-V specific, can should be applied to all the Hypervisors:

Do not pause, stop, or store the saved state of a domain controller in a virtual machine for time periods longer than the tombstone lifetime of the forest and then resume from the paused or saved state. Doing this can interfere with replication. To learn how to determine the tombstone lifetime for the forest, see Determine the Tombstone Lifetime for the Forest(http://go.microsoft.com/fwlink/?LinkId=137177).

Do not copy or clone virtual hard disks (VHDs).

Do not take or use a Snapshot of a virtual domain controller.

Do not use a differencing disk VHD on a virtual machine that is configured as a domain controller. This makes reverting to a previous version too easy, and it also decreases performance.

Do not use the Export feature on a virtual machine that is running a domain controller.

Do not restore a domain controller or attempt to roll back the contents of an Active Directory database by any means other than using a supported backup. For more information, see Backup and Restore Considerations for Virtualized Domain Controllers.

Note:

Never perform Hyper-V or any other Hypervisor Snapshot based activity for any of the Production System, Snapshot is meant for RnD environment, where they save time during testing environment for going back to the previous state, taking snapshot for production environment is never recommended, as it starts writing to Differential Disk (Hyper-V case), which reduce the VM performance also.

Note:
My blog and this blog post in particular, has no relation or liability on Microsoft, all the information provided here is to help, and after performing any of these below mentioned steps, if things go wrong on your side, then Microsoft or myself will not liable for that, do research and then perform these operation, as they are highly critical ones.

Steps for the recovering Domain Controller from USN roll-back condition:

1. First verify whether, you have the same problem, and after verification, further follow the process, You can use the Repadmin tool to make this determination. For information about how to use Repadmin, see Monitoring and Troubleshooting Active Directory Replication Using Repadmin (http://go.microsoft.com/fwlink/?LinkId=122830). If you are not able to determine this yourself, contact Microsoft Customer Service and Support(http://go.microsoft.com/fwlink/?LinkID=102491) for assistance, after you verified that you have the same exact symptoms of the problem, as specified above, then follow the process of demoting the affected DC

2. Go to other DCs, and from there see the hosted FSMO roles in your environment, and move (seize) all the roles, which are currently located on the effected DC, and after moving the roles to other DC, run Replication (repadmin /syncall /AdeP), now go back to the effected DC, and from there open CMD with dcpromo /forceremoveal to forcefully remove the Active Directory from the Server, and then you have to clean the meta-data about the old (removed) DC, because the DC was not able to send / receive replication, so the other DOMAIN CONTROLLERS are not aware the removal of the DC, so we have to take this DC out from the AD DB from others, for more information check this link.

3. Forcefully demote the domain controller. This involves cleaning up the domain controllers metadata and seizing the operations master (also known as flexible single master operations or FSMO) roles. For more information, see the Recovering from USN rollback section of article 875495 (http://go.microsoft.com/fwlink/?LinkID=137182) in the Microsoft Knowledge Base.

4. Once we have cleaned the meta-data of the DC, so now its our turn to re-install the ACTIVE DIRECTORY on the same VM, as normal, and once you will complete the installation, then you are back in the business!

5. As best practice, delete all former VHD files for the domain controller, so there will be no confusion.

More information on: Running Domain Controller in Virtualized Environment

VM based Domain Controller Replication Issues & USN-Roll back

I hope this blog post will help you to recover from USN roll-back situation, as it is so frustrating at times, because our DCs are like blood in our network.

Cheers!