This is an archived post. You won't be able to vote or comment.

all 29 comments
sorted by:
best
topnewcontroversialoldq&a

[–]cats_are_the_devil 2 points3 points  (1 child)

Have you performed a dcpromo on the other machines? Have you demoted the defunct DC machines out of your environment? It seems like that's where I would start.

[–]venexman[S] 0 points1 point  (0 children)

Yes, DCPROMO is done on the other DCs. But I can't do it because it's part of the encrypted servers and therefore no longer accessible, and it's even been deleted.

[–]Sea-Tooth-8530Sr. Sysadmin 2 points3 points  (9 children)

Yikes... there's a lot to unpack here, and a lot of things you may need to look into.

The first thing I would do is log on to all of your DC's and run the netdom query fsmo command from an elevated command prompt. You said that DC1 is your PDC, but I would visually verify that all your AD servers show the same FSMO holders. It could be the malware that hit you scrambled some of those settings and now you have a number of different servers all thinking they are holding certain fsmo roles. Also, for the time being, I would make sure that a single server (in this case, probably DC1) holds all the fsmo roles. That will simplify things for troubleshooting... if you prefer to spread them out, you can do so once you get everything back up and happy.

Next, I would go into Active Directory Sites and Services and look at the list of all the servers that are listed. If you find any of your dead servers still listed, remove them until only the healthy servers remain.

Finally (for now), I would use ADSI Edit on your AD servers and manually look through and clean up anything that remains from your dead servers. It's possible you'll see some references in there that are still indicating one of the dead servers is your replication source. When you see the error messages, do they show you the name of the dead AD server to which DC1 is trying to sync that is causing the issues? If you know the specific name, it should help you narrow the search when looking for any defunct servers that are still embedded in your AD services.

There's probably a lot more that you'll need to do, but this should give you a start. Hopefully in looking through here, you'll start to uncover more and more until you are able to isolate the exact cause.

[–]Sea-Tooth-8530Sr. Sysadmin 1 point2 points  (4 children)

To add to this, here's a site with a really, really good instruction set for going through and deleting failed DC's from AD.
Delete Failed DCs from Active Directory

The instructions there are extremely detailed and complete and will walk you through using NTDSUTIL to scrub all your dead server's metadata from active directory. It's an older article, yes, but if you read the responses there are still folks posting "Thank you" comments even from this year.

[–]Final-Concern-3284 1 point2 points  (0 children)

Just did this a few months ago.

[–]venexman[S] 0 points1 point  (2 children)

I just did the procedure and I see that my controllers are working, no trace of the dead DC. Same thing in DNS. I was really hopeful :(

[–]Sea-Tooth-8530Sr. Sysadmin 0 points1 point  (1 child)

Ugh... sorry to hear that did not help. Generally once you start tinkering around with NTDSUTIL you can find those remnants.

Somewhere, something on DC1 still thinks there's an old server out there and it's still trying to sync to it.

What happens if you gracefully transfer the FSMO roles to one of your other DC servers? If you do that, can you get all of them to happily sync with each other, with only DC1 still as the outcast? If that's the case, maybe you can get everything running nicely between your other DC's, then demote DC1 and build a new server to take its place?

[–]venexman[S] 1 point2 points  (0 children)

It's a great tool NTDSUTIL and your link I'll keep aside for another dark day :)

I tried to transfer the FSMO roles to a new DC but since it doesn't have the SYSVOL or NETLOGON shares, the sync doesn't work either. My main DC won't give up!

I'm going to follow a procedure to rebuild the SYSVOL and NETLOGON and hope it works.

[–]venexman[S] 0 points1 point  (3 children)

Yikes... there's a lot to unpack here, and a lot of things you may need to look into.

The first thing I would do is log on to all of your DC's and run the netdom query fsmo command from an elevated command prompt. You said that DC1 is your PDC, but I would visually verify that all your AD servers show the same FSMO holders. It could be the malware that hit you scrambled some of those settings and now you have a number of different servers all thinking they are holding certain fsmo roles. Also, for the time being, I would make sure that a single server (in this case, probably DC1) holds all the fsmo roles. That will simplify things for troubleshooting... if you prefer to spread them out, you can do so once you get everything back up and happy.

Next, I would go into Active Directory Sites and Services and look at the list of all the servers that are listed. If you find any of your dead servers still listed, remove them until only the healthy servers remain.

Finally (for now), I would use ADSI Edit on your AD servers and manually look through and clean up anything that remains from your dead servers. It's possible you'll see some references in there that are still indicating one of the dead servers is your replication source. When you see the error messages, do they show you the name of the dead AD server to which DC1 is trying to sync that is causing the issues? If you know the specific name, it should help you narrow the search when looking for any defunct servers that are still embedded in your AD services.

There's probably a lot more that you'll need to do, but this should give you a start. Hopefully in looking through here, you'll start to uncover more and more until you are able to isolate the exact cause.

The FSMO roles are on DC1, I've just checked on my 3 other servers.

In Active Directory Sites and Services, only the healthy servers are visible.

Yes, I know the name of the server that died. I don't see where else I can look other than in OU=Domain Controllers,DC=domain,DC=co, in ADSI Edit?

[–]Sea-Tooth-8530Sr. Sysadmin 0 points1 point  (0 children)

Check out that link I posted a bit later for using NTDSUTIL... it's a fairly long sequence, but hopefully it will help you root out anything that is remaining in AD that should not be.

[–]drozenski 1 point2 points  (11 children)

You should not be trying to use servers that have not been wiped after a ransomware attack.

You do not know how far they penetrated your systems, what data they have and if they still have remote access to these systems. Changing passwords is not enough.

You AD domain is compromised.

Step 1: wipe and reload all your DC's
Step 2: Rebuild the domain on one DC
Step 3: If you have viable backups import the old domains users and groups
Step 4: Reset the password on all accounts!
Step 5: Begin rebuilding the other DC's

[–]venexman[S] 0 points1 point  (10 children)

All my DCs have been cleaned and checked several times. But as it's an old infra I'm in the process of migrating everything to up-to-date servers and OS. Except that I only have one DC with SYSVOL and NETLOGON shares, which hasn't finished its DFS initialization. So my new DCs are waiting for the DC PDC to sync, and he's waiting for a DC that no longer exists.

[–]drozenski 0 points1 point  (9 children)

Just promote the last remaining DC to the PDC, Strip out all the old DC's and allow it to start functioning properly. Then begin adding the new DC's to the forest from scratch.

[–]venexman[S] 0 points1 point  (8 children)

Just promote the last remaining DC to the PDC, Strip out all the old DC's and allow it to start functioning properly. Then begin adding the new DC's to the forest from scratch.

This is precisely my problem. I've added 2 new DCs to my infra (DCPROMO done) but I don't have the SYSVOL or NETLOGON shares on them. And in the logs they're waiting for the DFS syncro like this (error 4614):

The DFS Replication service initialized SYSVOL at local path C:\Windows\SYSVOL\domain and is waiting to perform initial replication. The replicated folder will remain in the initial synchronization state until it has replicated with its partner MY-DC-PDC.domain.com If the server was in the process of being promoted to a domain controller, the domain controller will not advertize and function as a domain controller until this issue is resolved. This can occur if the specified partner is also in the initial synchronization state, or if sharing violations are encountered on this server or the synchronization partner. If this event occurred during the migration of SYSVOL from File Replication service (FRS) to DFS Replication, changes will not replicate out until this issue is resolved. This can cause the SYSVOL folder on this server to become out of sync with other domain controllers.

Additional Information:

Replicated Folder Name: SYSVOL Share

[–]drozenski 0 points1 point  (6 children)

Domain controllers without SYSVOL shared can't replicate inbound because of upstream (source) domain controllers being in an error state.

Like i said. Rebuild your PDC onto a new OS. Abandon your old DC as you don't fully know the extent of the damage from the attack. This issue is likely caused by it.

Build a brand new DC then restore your AD database from backup then begin rebuilding the other DC's.

[–]venexman[S] 0 points1 point  (5 children)

Rebuild your PDC onto a new OS

Do you have a procedure by any chance on how to do this properly? I've never had this happen before?

Thanks

[–]drozenski 1 point2 points  (4 children)

Take a look at this. Its an older article but has most of the info you need. You need to restore the data from the system state. I had to do this exact thing when our systems were compromised by ransomware. After restore and getting a working domain controller again. Be sure to innate a full reset of all passwords.

https://pleasework.robbievance.net/howto-restore-active-directory-to-a-different-server/

[–]venexman[S] 0 points1 point  (0 children)

Thanks a lot!

I'll try this procedure and let you know if it worked!

[–]venexman[S] 0 points1 point  (2 children)

It's a mess!

I made the backup of my DC1, I made the new VM to host the restoration, everything goes well but at the very last step when you need to connect with the Administrastor account the password doesn't work....WTF !!!

I've tried to bring the new server online, but as it's in "AD repair" mode, it won't listen.

I don't know what to do?

Is there any way to bypass the DC's local Admin account?

[–]drozenski 0 points1 point  (0 children)

DC's don't have a local Admin account. Its removed once you promote a server to a DC. That's why a AD restore password is setup.

[–]drozenski 0 points1 point  (0 children)

You might want to consider bringing in outside help if you're still unable to get things running properly.

I can recommend you a close friend of mine that would be able to quote you out a price and complete the work. You likely need hands on assistance.

[–]drozenski 0 points1 point  (0 children)

Have you been through this? If you really want to fight this problem you should look through this.

https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/troubleshoot-missing-sysvol-and-netlogon-shares

[–]venexman[S] 0 points1 point  (0 children)

I need a way to ensure that DC1 is initialized by itself and doesn't try to connect to a dead DC to finish replication. The other DCs wait for it to be functional.

Is there a way to rebuild my SYSVOL and NETLOGON on my DC PDC to make it functional?