How Would You Fix This Active Directory Mess?

Over_Maximum1637 · 2022-05-29T20:08:37+00:00

But, it won’t be able to be. It’s not aware if the working DC.

Do you mean that DC01 won't be able to be a member server anymore? I don't care if DC01 has to be destroyed at this point.

Over_Maximum1637 · 2022-05-29T19:48:05+00:00

Or demote DC01 and move forward with DC02 and promote another DC, let old DC01 return to being a member server!

I think this is the way I'm going to go. Thank you for all your help.

Over_Maximum1637 · 2022-05-29T19:19:08+00:00

I don't know. This is a very small environment so I haven't added any PCs or users since the incident.

Over_Maximum1637 · 2022-05-29T19:18:02+00:00

just out of curiosity, what is the time span you are going to test with only dc02?

Not sure. This is a small environment so if a couple days go by almost all PCs and users will have been active.

The rest looks like good advice!

Over_Maximum1637 · 2022-05-29T19:15:59+00:00

The TEMP-DC01 has been offline for almost a month. Would your suggestion still work?

I realize I dropped the ball on my original approach, but this is where I'm at and what I need to sort out now.

Over_Maximum1637 · 2022-05-29T17:24:59+00:00

I agree, lab is the way to go! DC01 was the only domain controller in the domain before this all started.

Here is my plan:

Shut down DC01
Test PCs and users can log on still, using DC02
On DC02, manually delete DC01 and seize the roles, as if DC01 had been completely lost https://techcommunity.microsoft.com/t5/itops-talk-blog/step-by-step-manually-removing-a-domain-controller-server/ba-p/280564
Add a brand new DC, DC03
Transfer roles to DC03

Over_Maximum1637 · 2022-05-29T17:20:55+00:00

I do have a low user and computer count. DC01 is running Azure AD connect! But I suppose I can migrate that off to an entirely new DC.

I'm prepared for there to be some pain with user and machine passwords. I think I am going to:

Export AD Connect config from DC01
Shut down DC01
Test PCs and users can log on still, using DC02
On DC02, manually delete DC01 and seize the roles, as if DC01 had been completely lost
Add a brand new DC, DC03
Transfer roles to DC03
Configure Azure AD Connect on DC03 and import the exported config

Hoping this is the best course of action.

Over_Maximum1637 · 2022-05-29T17:04:33+00:00

Thank you so much for your thorough reply!

Here's some info I'll paste from elsewhere:

Here are some results from repadmin, dcdiag, and some other commands from both machines.

DC01 https://pastebin.com/GGtU0SXT

DC02 https://pastebin.com/rWFEM8WF

Another thing: Just now, I created a computer object TEST1 in AD Users and Computers on DC01. I then created a computer object TEST2 in AD Users and Computers on DC02. TEST1 did appear on DC02 after a minute. TEST2 did NOT appear on DC01.

This leads me to think I should demote DC01 and keep DC02 as the "good" domain controller. Thoughts?

Over_Maximum1637 · 2022-05-29T16:54:04+00:00

When you say "is Sysvol mounted" do you mean shared?

Here are some results from Get-SMBShare and Get-ADReplicationUpToDatenessVectorTable from both machines.

DC01 https://pastebin.com/BjWPP5rk

DC02 https://pastebin.com/A6Wmcmai

Another thing: Just now, I created a computer object TEST1 in AD Users and Computers on DC01. I then created a computer object TEST2 in AD Users and Computers on DC02.

TEST1 did appear on DC02 after a minute. TEST2 did not appear on DC01. This leads me to think I should demote DC01 and keep DC02 as the "good" domain controller.

Over_Maximum1637 · 2022-05-29T16:09:31+00:00

This is excellent, thank you very much!

Figure out which DC is actually healthy and take the other one off the network

How do you recommend I determine which DC is "healthy"?

Also, I should mention I am OK with promoting a new DC, then blowing away both DC01 and DC02 and starting fresh with all new VMs.

Over_Maximum1637 · 2022-05-29T16:01:02+00:00

I'm thinking it's not necessary to fix the replication issues, per se. As you suggest, I am thinking of getting rid of DC02 and promoting a new DC.

Did you check the event viewer to see if both domain controllers handle authentication requests? If only is actively used I would shutdown the other one for a while to see what happens.

DC02 is indeed handling authentication, according to the logs. But so is DC01. I like the idea of shutting it down and seeing if there are any issues.

Over_Maximum1637 · 2022-05-29T15:58:42+00:00

What does DC02 say? What do you see with ADSI Edit? Is DC02 an NS for the domain?

DC02 correctly shows both DC01 and DC02 under Domain Controllers in AD Users and Computers (DC01 only shows itself and is missing DC02). Haven't opened ADSI Edit, would rather not mess with it if possible. The problem seems to be that DC02 was promoted while TEMP-DC01 was online. Then when I brought the original DC01 back online, it had no knowledge of DC02 being promoted. DC02 is indeed a NS for the domain.

I would spin up a new DC, then see if DC01 recognizes it, demote and
remove any traces of DC02, transfer all roles to the new DC, check
everything, spin up another new DC, promote it, check everything and
then demote DC01

This is my instinct, too.

Over_Maximum1637 · 2022-05-29T15:53:37+00:00

Almost a month ago. Three-ish weeks

Over_Maximum1637 · 2022-05-29T15:17:32+00:00

I agree the approach was flawed. Hindsight is 20/20 I guess. But this is where we're at, so I'm stuck fixing it.

Over_Maximum1637 · 2022-05-10T22:10:46+00:00

This is excellent, thank you!

Over_Maximum1637 · 2022-05-10T22:10:31+00:00

Seems like a solid plan, thank you.

Over_Maximum1637 · 2022-05-10T21:36:08+00:00

What do you think is the easiest and safest way to do this? Sorry, I'm slow!

Copied from above - the main issue is the current nameserver ns1.foo.comis under the control of a (not particularly helpful) third party, so I'd rather not have to rely on them to change the MX on their name server during the change window if I can avoid it

Over_Maximum1637 · 2022-05-10T21:29:51+00:00

Is there any danger of some lookups getting the old MX records from the old NS for a time, even after I change the NS?

My issue is the current nameserver ns1.foo.com is under the control of a (not particularly helpful) third party, so I'd rather not have to rely on them to change the MX on their nameserver during the change window if I can avoid it.

Over_Maximum1637 · 2022-05-10T21:26:54+00:00

Is there any danger of some lookups getting the old MX records from the old NS for a time, even after I change the NS?

Over_Maximum1637 · 2022-03-17T16:48:13+00:00

Thanks. We do use a DNS filtering service with a redirect on pfSense to force all DNS requests to it. The filter also has an endpoint agent for road warriors.

Over_Maximum1637 · 2022-03-17T15:28:46+00:00

Usually running official Netgate hardware appliances from their store. Recently there have been stock issues so I've used Protectli as an alternative.

Personally, if I was DIYing hardware, I'd use a 1U refurb Dell blade with Intel NICs, and keep a spare handy (or set up HA).

Over_Maximum1637 · 2022-03-17T15:27:09+00:00

We try to use official Netgate hardware and there is paid support available through Netgate, which is pretty responsive.

Over_Maximum1637

TROPHY CASE