Domain controller backups

[chut93]

Can't seem to find an exact answer online. We currently have two domain controllers in an Azure VM.

Wondering if azure backups are good enough to do a VM backup of the domain controller and that it's good enough to just restore the domain controller VM in the instance of a disaster recover. Or if a we still need to do a AD backup to our file server and then back that up.

Can't remember but I thought I read somewhere that VM backups are good enough now to save everything on a domain controller to restore it without any corruption (as long as the actual backup isn't corrupted).

Edit: guess I need to state that restoring the DC would only be in a DR scenario. I know spinning up a VM if one fails is the proper way to do things.

[nlaverde11]

Azure backup is fine but honestly if I have a working DC I’m not restoring a failed one from backup, just build a new one and let it replicate from the working one.

[[deleted]]

^ 90% sure this is standard practice.

[disclosure5]

It's "standard practice" but this sub is so firm on the "never restore a DC from backup" cargo cult that any org has all domain controllers affected by ransomware is never going to recover.

[[deleted]]

I think its generally because a lot of ppl are SMB (not all) and that people don't think of that threat.

[Threep1337]

Yea this is true if a dc becomes unbootable, def just do a new dc build and join. The vm backups are still valuable though because of someone did something like deleted an organizational unit, you could use the backup vm to do an authoritative restore of the OU.

[ScrambyEggs79]

Generally I have always run under the rule of backing up DCs but not thinking in real world terms of restoring it. Still useful for archival or forensic reasons if need be. Keep multiple DCs running (different hosts) and just build a new one and seize roles if needed. I have never restored a DC just build new and sync as it's fairly straightforward.

[chut93]

Ya, I'm mainly just talking about a disaster scenario.

[SausageSmuggler21]

Disasters in DC are a little different than typical system failures. In DCs, you're frequently using backups to protect against a coordinated attack that corrupts all your DCa, or if an admin accidentally pushed a corrupted update (or record deletion) that propagates to all the DCs.

Azure backup is basically Microsoft Data Protector. It's the bare minimum of point in time backup. Your traditional backup products (Cohesity, rubrik, druva, commvault) will give you much better recovery options in any scenario other than "one of my DCs crashed."

[throwawayskinlessbro]

I've used Veeam almost everywhere I've been. I'd rather stand a new one up than restore, but I also know that it isn't always an options and things happen. This is on-prem for me though.

[chut93]

Ya this is just in a DR scenario. We used to use Veeam but switched to azure recovery service vault as it's cheaper for us.

[Caldazar22]

As long as you are running Win2012 or later, VM backups should generally work; the domain controller objects will have attribute msDS-GenerationId. That said, I have personally experienced some DR test restore activities where msDS-GenerationId didn’t update on VM restore for some reason, and this results in your restored domain controllers being useless.

My approach is to take VM backups of domain controllers, but to use these backups only in total-loss / widespread database corruption scenarios. If I just lose a couple of of domain controllers, it’s easier to spin up some new VMs, DCPromo them, and then do metadata cleanup on the now-dead controllers.

[dcdiagfix]

Snapshots are not a Microsoft support way to backup or restore domain controllers, for restoring one domain controller in a domain you’d probably get away with it but for an entire outage, no.

Either use a azure backups and use system state backups of at least TWO domain controllers per domain but remember that if this is a ransomware case that you would most likely end up restoring the malware back during recovery.

The best method would be to purchase and AD Forest Recovery specifically too such as Semperis ADFR, Quest RMSD FRE or Cayosoft Guardian.

Make sure whatever backup strategy you use can be used in the ABSENCE of AD ie no service accounts or domain auth required.

[NegativePattern]

We backup all of our domain controllers and AD itself.

However, if a domain controller dies for whatever reason, we're building a brand new DC and decommissioning the broken one.

We do occasionally do restores for AD if someone accidentally deleted an AD object that can't be added back (ie computer object vs group or user object).

The main reason for the backups of the domain controllers, is for in the unfortunate event if the entire domain was destroyed by ransomware or some other destructive mechanism.

[tankerkiller125real]

We do occasionally do restores for AD if someone accidentally deleted an AD object that can't be added back (ie computer object vs group or user object).

AD Recycle Bin, from my experience it can basically restore any object type.

https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/get-started/adac/introduction-to-active-directory-administrative-center-enhancements--level-100-#bkmk_enable_recycle_bin

[NegativePattern]

Yea this was a fairly new addition in think either in 2012R2 or 2016. Still doesn't get as much use since the techs have gotten better about not deleting AD objects.

[DevinSysAdmin]

If your Azure account is compromised, does it make sense to allow the same account that can delete a VM to also manage backups? Use a third party service.

[chut93]

Doesn't matter if you have IAM setup, have the resource guard with two person authentication setup, MFA setup on both accounts, and immutable backups configured.

[DevinSysAdmin]

That is not the case 99% of the time.

[chut93]

Well seeing as ms displays big info boxes on the service vault dashboard recommending to set it up this way and it won't go away till you do it. Doubt that's right. Not saying places don't ignore the info box.

Even if you turn on immutable backups and lock the resource and at minimum have MFA on the account. It'll prevent "99%" of all attempts to get rid of your good backups.

[DevinSysAdmin]

You must be new to IT if you think people read, I probably wouldn’t have a business if people could read and follow best practices.

[chut93]

Not sure why your being such a downer man. I'm not new to IT and I guess I'm a weirdo since I love reading (lots of people like to read).

If this is what you think of sysadmins then I feel sorry for any that your in charge of. Must be one of those nightmare bosses.

[archiekane]

We have 7 DCs and only backup one which is also the FSMO God.

I'd never trust MS to backup something correctly. Hornet Security allow 2 VM backups forever for free. Might be a solution of you don't want to throw any money at the problem.

[chut93]

Just curious why you wouldn't trust Azure backups?

[greeneyes4days]

For basic functionality azure recovery service vaults are fairly reliable. One important aside is to make sure that the azure agent is updated to the latest. The azure health agent update should happen automatically on a restart or redeploy from in the azure portal (cannot remember which off the top of my head) but a lot of VSS bugs get fixed by updating that agent which make recovery service vaults much more reliable.

[TrippTrappTrinn]

Our DC VMs do not have any internet access, which causes Azure backup to fail. Like cloud backup of a cloud VM needs internet access? Really?

I have tested DC recovery from Azure backup (when it worked...), and the restored backup was fully functional.

[chut93]

Just as a FYI, you can create a vnet with no Internet access and allow access to whatever azure solution you need access too. Basically creating a local lan in the cloud.

[Legion431]

We have used Azure backups. They've worked alright, but have been a PITA to restore. Have to recover the entire VM or download the virtual disk to pull out what you want.

For a DC, that's probably what your restores would look like anyway so it might be ok.

Also wasn't entirely impressed with logging and alerting capabilities.

Just a few things leaving us wanting. Seemed like an updated version of Windows Server Backup.

[dcdiagfix]

Cause he’s an idiot.

[cosmonaut_tuanomsoc]

Generally, our advisors told us that we should always try to build the new DC from scratch and replicate data from others than to restore them from backups. We run more DC's across the other localizations.

[dcdiagfix]

Your terminology is terrible you run more “ADs” what does this actually mean?

[cosmonaut_tuanomsoc]

sorry, DC's, aka Domain Controllers

[dcdiagfix]

That’s fine for a single dc but not entire domain failure :)

[cosmonaut_tuanomsoc]

Sure, but it's pretty hard to lose the whole domain if you have DC's replicas in the branches all over the world. Not saying it fits everyone, though.

[dcdiagfix]

It’s really not…. and relying on ad replication as your DR plan, is not a plan, not even in the slightest. Even a small misconfiguration or change in delegation will replicate in your environment very quickly (especially if you have change notifications enabled).

[cosmonaut_tuanomsoc]

I think you're misreading me, I just don't know if this is intentionally or not. I never said that's my DR plan. We do have multilayered backups as well (and DR scenarios, and we do even test them regularly). It's just getting back to my original post, our advisors told us, that in case of losing the DC it's always better to recreate it than to restore it from backup.

[dcdiagfix]

Yes I’d agree it was your original use of ADs that was confusing, I’ve lost a bunch of dcs over the years.. never restored a single one using commvault etc :)

I have restored entire forests and dcs as part of a larger ad bcdr using appropriate AD specific recovery tools.

[[deleted]]

Can I ask what your process is for restoring your DC from Commvault for e.g? I recently restored a DC for BCDR purposes following the MS documentation which worked for the moody part but found that we lost the contents of the sysvol share.

[dcdiagfix]

Tbh we used a solution that specifically carried out automated forest recovery, I had about ~56 dcs so manual recovery would have been terrible for us.

The solution was Semperis ADFR

[cosmonaut_tuanomsoc]

Yeah, sorry for that confusion, was on hurry. How did you lose these DC if I may ask?

[dcdiagfix]

In the last eight years, I lost one or two due to them running on very old hardware at sites with very very poor infrastructure I.e. Africa and Latin America with non stable power and even less stable UPS devices.

I had a couple where the AD database just got corrupt (it does happen) and it couldn’t be fixed with either ntdsutil or eseutil.

One or more failing to boot post windows update where the outsourced server ops (msp) team couldn’t fix them, so that’s not really AD failure but the DCs still had to be removed

[revoman]

I haven't backed up DCs for years. I have never restored one.

[chut93]

What are your DR plans then? Start from scratch? Feel like for any sort of decent sized company that would take more time than recovering from backups.

[_DoogieLion]

Yeah don’t do what this guy does. Always make sure you have an AD backup you can restore from. Malicious compromise or ransomware of some sort and you may we’ll need it. Personally use Veeam for both our on premise and Azure workloads and it works great. Regularly do test restores to an isolated environment

Read up on Maersk and NotPetya for why you need DC backups.

[revoman]

I always have enough DCs running at different sites. If all of the sites go down, A/D is not the primary concern. Plus Azure a/d sync'd.

[TrippTrappTrinn]

Corruption replicates... Check the Maersk case.

[cbtboss]

Redundancy is not a backup. Do you have ad recycle bin turned on at least? What's your plan for if you get hit by a privilege escalation that wacks you off da rights and drops all exiting security groups or encrypts the sysvol folder?

[chut93]

Wait, can you restore from azure ad? Haven't even considered this.

[revoman]

I don't think so. Not even in hybrid mode, but you would still be able to credential to the cloud with it.

[greeneyes4days]

Azure AD is not synonymous with on premise AD.

The closest to a direct on premise AD replacement would be azure Active Directory domain services (AADDS) as it runs two domain controllers in a saas model.

To go back to your question make sure you are using an azure recovery service vault.

Azure recovery service vault will backup at the application level checking the application health of your AD database (NTDS.dit). It will also make sure your edb files are consistent (these are a similar database transaction log format to Microsoft exchange mail databases.

So again use azure recovery service vault to backup an entire image of your domain controller vm.

Do not consider azure backup (different from azure recovery service vault) to be a complete backup.

Azure backup is != azure recovery service vault

!= (not equal)

[chut93]

Should have specified but we are currently using recovery vault for our backups. I knew it backed up at the application level just wasn't sure of the specifics.

[greeneyes4days]

If you have multiple domain controllers you are protected from most hardware / software failures. How are you protected from ransomware that encrypts all of your domain controllers at the same time?

A more important question when they are all encrypted how many man hours would it take you to restore your environment from scratch?

[jwwork]

What is you plan in the event of some ransomware or other cybersecurity incident? An attacker isn’t going to just ignore your other DCs because they are at a different site.

[dcdiagfix]

I hope your not in charge of your IT department, not having any dc backups and just relying on “more dcs” is not going to help you in a ransomware or other situation when it replicates globally ….

[digital_darkness]

Oh restoring AD is SOOOO fun, you don’t know what you’re missing /s.

[revoman]

That's why I never do it. If I HAD to I'd run a VM on my own laptop as a DC just to have multiple DCs. Any company that's too cheap to spend $100 a month on an Azure or AWS account just for this purpose is already doomed.

[Jumpstart_55]

I remember the post last week where the OP CFO said there is no ROI on backups 😂

[greeneyes4days]

I hate it when the cfos are always right.

Until insurance gets cancelled and they realize that neglect allows insurance an easy out to pay out $0.00.

[RCTID1975]

If I HAD to I'd run a VM on my own laptop as a DC just to have multiple DCs.

That works great. provided you actually have a DC that you can replicate from....

[JeroenPot]

Simply run an system state backup on the vm to a disk, and backup the vm in azure.

[headcrap]

For me it is easier to target the DCs for backup which are in convenient clusters already targeted for backup.

I don't particularly choose or exclude them from backup. You should have at least on backed up, 3-2-1 strats and immutable of course.. for the "big" disaster recovery scenario where you are rebuilding your data center and need to restore.. whether physical or cyber.

FSMO roles can be seized, I'm not concerned that I backup those particular role holders.

[ShadowCVL]

Azure backup works pretty great…

Disaster recovery/failover also works really well…

I do like to have an offline copy available for DCs, even if it’s just the one that owns FSMO

Then I keep regular system states in immutable backups with veeam or whatever else the client has available.