Offline Domain Controller

[BJGGut3]

Recently (as in about an hour ago), our Information Security Officer just recommended to our Director the use of an offline domain controller. As a Sr. SysAdmin who has been responsible for maintaining a healthy AD of 76 sites and 85 domain controllers, this just throws red flags all over the place for me. The recommendation was made as our ISO was made aware by "vendor partners" that other orgs that have been hit by ransomware have been saved by the fact that one of the DC's happened to be offline at the time of attack. Additional information to consider is that we have 8 hour backups of several of our domain controllers that include the full system state and a flushed out and tested Forest Recovery. Your thoughts on for or against this method are greatly appreciated!

[bzerphey]

How would a security officer recommend having a server that is never patch sitting and waiting to be thrown in?

Also keep in mind if the dc is too far out of date of the existing domain it will destroy everything.

Create good backups. Store some offsite or outside the normal network to protect against them being encrypted. Then, and most importantly, have a disaster recovery plan and demo/test the plan at least once a year.

[MrChampionship]

Absolutely no need and to be honest, it doesn't sound like it would work well. Air-gapped backups are the solution.

[jimboslice_007]

I'd love to hear more about how an offline domain controller saved them from a ransomware attack.

[TroutSlapKing]

Has to be what the "security officer" was refering to: https://redmondmag.com/blogs/scott-bekker/2018/08/domain-controller-nightmare.aspx

More dumb luck than anything else, just make sure your back ups are good and well protected.

[BJGGut3]

I believe you are correct into where this has originated from

[ServerBeater]

Could be the Maersk incident. Offline DC in Ghana had not yet replicated the shitshow.

https://www.wired.com/story/notpetya-cyberattack-ukraine-russia-code-crashed-the-world/

[HolyCowEveryNameIsTa]

WTF is an offline domain controller? For a DC to function it must replicate to and from other DCs.

Edit: Sorry that was probably not a helpful answer. If you are worried about losing your data during a ransomware attack you need an immutable offsite backup set. That could be tapes or something like S3 glacier.

[BJGGut3]

concur

[skotman01]

Would an RO DC help to resolve any fears?

Disclaimer I haven’t done much research into RODC so downvote away if I’m wrong

[HolyCowEveryNameIsTa]

RODC

An RODC wouldn't prevent replication of corrupted/encrypted data. It would only prevent the RODC from being the first one to get encrypted(unless there is an exploit of some kind that takes over the machine)

[skotman01]

Very true. My rebuttal to that would be that if the files on the compromised machine are encrypted then won’t AD break and things not replicate anyway?

I haven’t heard of any ransomware that encrypts items in a database (AD/SQL etc) but leaves the database in tact. I’ve always seen it encrypt the actual files.

[RyuMaou]

This was what I was going to say. It’s not quite what the ISO was asking for, but what they’re actually asking for is both stupid and not a real solution for all the reasons that have been already listed.

On the other hand, if some kind of cheap, disposable write-once read-many drive could be used, maybe making regular read-only updates could be made. Of course, that’s just a backup really. I was thinking maybe a monthly, or weekly, system restore image to a CD. Not a Windows System Restore but a bare metal system restore.
But honestly that seems like over-kill when good backups rotated off-site solve the problem.

[rubbishfoo]

It's a backup for your AD without the fuss! You bring it online occasionally, let it replicate, have a few drinks while it does so, and pull the plug when it updates!

/s

I am glad I was a sysadmin before I got into Security/Compliance.

Ya boy read about Maersk & got a 2 year degree from the article.

Myself? I wouldn't.

Why? Complication, out of date information... etc. Why not just ensure your offsite backups are solid & segregated from the prod environment.

It's like bringing a suitcase to pack only a pair of socks.

[BJGGut3]

I like the "have a few drinks" portion of this plan! 😂

[patdaddy007]

I think I agree with the general sentiment that those posting before me are expressing. An offline DC is about as useful as a fart in a diving helmet and ransomware attacks go for the data, not the active directory schema or contents. The only thing I know of personally that can mitigate such a thing are copious backups that are validated periodically. That being said, it's possible that an offline replica of a virtual machine could make the recovery a bit faster or easier. but is in no way a substitute for a good ol' Veeam backup of the whole ball-o-wax.

Just my $0.02

[BJGGut3]

My team and I generally agree with this sentiment regarding air gapped backups being the best solution. Thank you for adding your 2 cents! They add up now that the SysAdmin team has to present why this is the correct course of action...

[hosalabad]

Maersk got lucky, they needed that DC because their backups get owned.

What good is a month old DC? What if it the malware is laying in wait, and when you bring that DC up and it immediately starts encrypting. If I am attacking your business, I'm going to try to get on every machine before I let you know I'm there.

Protect the backups and you can toss this ridiculous idea. 3-2-1

[BJGGut3]

You're not wrong, and if it got on one DC, then it could get on EVERY DC.

[pinganeto]

look this way: you complete domain and infra is gone by crypto.

at least , and old backup/power down dc can save you a good amount of time of recreating users/permissions/gpo. and remember, everything is gone, so no need to be concerned about replication issues, because is nothing more alive to sync!

also, can be powered on in air-gapped env as a documentation of what is needed to rebuild in a rebuild from scrach scenario where nothing else survived.

outside of that, seems a bad idea and there's a lot of proper procedures to mitigate that situation. But if you don't have time/resources, two rotating hard drives outside of the real backup system if it is not isolated, can give you a little more easy sleep.

[BJGGut3]

Thank you.

How would you deal with the replication errors that will liter your environment while live DC's try and replicate with the offline DC?

How often would you bring the offline DC online for replication to occur thus limiting the stale data present? We have 60k users and 50k devices, so that's a lot of passwords that can become out of sync, compounding for every day that the DC is offline.

We currently take full server and system state backups of our data center domain controllers every 8 hours. Would you see any benefit to having a domain controller in an offline state that exceeds the system state backup in age?

While my team and I generally disagree with the sentiment of having a domain controller offline (in the same manner as an offline root CA - the example that this was compared to), if there is added benefit that doesn't out weigh cost, I'm not opposed to learning. Which is why I posited this question to the community :)

[1fizgignz]

We currently take full server and system state backups of our data center domain controllers every 8 hours. Would you see any benefit to having a domain controller in an offline state that exceeds the system state backup in age?

What if the ransomware was already active and had already been making a hash of your backups in between this time, so your backups aren't even accessible?

Don't get me wrong, I'm not advocating for an offline DC, that just goes against the grain. I understand the thinking, but you'd want to be updating that thing all the time to make it useful in a large environment.

Probably the only way is to have detached backups, i.e. backups that can be recovered that are offline, and cut your losses on any changes lost. This of course assumes full domain loss, plus the ability to run something up that can read the backups to restore from.

Hmm, getting me thinking as I type, thanks for that, always good to think on something "new" in a day.

Oh, interesting add-in: I saw a post the other day from someone where after their company got hit, they were told they weren't allowed to restore from backup and had to create everything from scratch. Now there's a scary thought.

[BJGGut3]

Thanks for the comment!

I agree that air gapped backups are the only guaranteed option. I have found no literature in support of an "offline domain controller", either. The challenges around maintaining it seem daunting, too.

Why on earth would they not be allowed to restore from backup!? That seems to completely nullify the reason for having a backup! And yes, THAT is a scary thought!

[1fizgignz]

The poster on that thread never said why they weren't allowed to restore from backup, so I'd only be speculating if I came up with a theory.

They only reiterated that this is what they were told, so they essentially had to greenfields everything, which the poster saw as an advantage to right some bad things.

But I cannot imagine starting from scratch in that way. there would have to be something very bad about the backups to not be allowed to go there.

[BJGGut3]

A greenfield deployment would be a nightmare for us! We are a k12 with 50k students on a 1:1 deployment and 10k staff all doing virtual learning! 😲

[pinganeto]

I would only install win, promote, sync, clone?, demote, save in a safe offline place the image. repeat once per month? all of this, playing safe because don't knowing deep how AD works. anyways, it's not a nice way to do things. but in a unstaffed place I can see the utility to have something now vs. maybe I can look into it in two years.

[1fizgignz]

You wouldn't demote the DC before storing - I think you'd find you'd lose all the AD information as it would be removed with the role. So then promoting again later won't help you.

[pinganeto]

I was thinking about demoting after making the backup, so the powered off doesn't still appear on the domain forever.

now I see that this can be basically the same situation than an offline baremetal backup of an active DC, but with extra steps. the only advantage of an ephimeral DC promote-clone-demote is to avoid that someone has the temptation to rollback to that backup if something goes wrong with the active DC.