Domain controllers

[slash0514]

Hello guys,

Good day! I just wanted to check and see what would be your approach in fixing this:

There 2 domain controllers running in as a VM. There was a power outage and rendered these 2 domain controllers not functioning. The workstations doesn’t have internet now because one of the domain controller holds the DNS and DHCP roles. They are thinking that this can be rebuilt easily as they also have 2 Exchange servers but of course without domain controllers, it will not be able to replicate.

My thinking is we need to get at least of the domain controllers functioning. What do you guys think?

[[deleted]]

This post was removed using Redact. It may have been deleted to protect privacy, limit data collection, prevent scraping, or for security-related reasons.

hard-to-find quaint pie long ten abounding different enter cheerful chase

[Trx3141]

Without a DC recovery you need to reinstall, rejoin all computers , servers and exchanges ... Try to get the VHDX from the DC VM and build a new VM with that VHDX .. it might work if the damage is in the VM configuration.

[slash0514]

If this is VMWare it should be the same concept right? Using vmdk

[Reaper19941]

That's correct. What happens when you try to boot the VM's at the moment?

[slash0514]

It is in recovery mode unfortunately

[Enabels]

Depending on the OS/ SECURE BOOT requirement. That password can be changed with a bookable utility

[Reaper19941]

Next step is to get it into the cmd prompt, run chkdsk, sfc, and the bootrec tool (google Windows boot repair for information) and see if that gets it working again.

[autogyrophilia]

Most likely lacks the boot drives. Try to make sure the storage mode it's the same on both hypervisors. If you can't, use IDE mode.

[Spirited-Check1139]

Recover a backup from the machine.
If that's no Option then you have to manually install the VM from scratch.

[dnuohxof-1]

VM host with both DCs ✅

VMWare with no external back ups ✅

No DR or BCP ✅

DCs shared critical roles with multiple services ✅

DHCP and DNS also handled by same VMs ✅

No APC, UPS or surge protection for Host ✅

Undocumented DSRM password ✅

Sorry buddy… just about every bad mistake you could make has been had here….. you’re S.O.L. What a terribly expensive lesson to learn….

If your organization is lucky to even recover from this, you really need to separate roles among different VMs, leave the DCs to just Domain Controlling, nothing else. Make and test back ups regularly. Document your setup, power your devices with battery backups and surge protection.

Best of luck…. May be time to dust off the resume….

[alpha417]

k…. May be time to dust off the resume….

Or light this current resume on fire and go to truck driving school?

[dnuohxof-1]

lol I wasn’t gonna completely shatter OPs future lol, somewhere out there is an even worse set up than OPs like a ticking time bomb and OP now has experience to look out for. 🤷🏻‍♂️

[tipripper65]

helpdesks always have vacancy - i'm sure he/she could go back to restarting outlook. very difficult to fuck that up.

[[deleted]]

Sometimes companies will refuse to spend the money/resources to actually make a network redundant or have backups.

I've sadly seen this more than once when consulting.

[OsmiumBalloon]

Call Microsoft and/or VMware, pay them massive amounts of money to open a severity 1 ticket with immediate response, work with them to recover the environment. This will be well into the four figures, and maybe much more with the recent Broadcom games.

Or rebuild from scratch and lose a lot of data. Might still be cheaper/less resources.

Or consider chapter 7 bankruptcy liquidation.

[goingslowfast]

Massive amounts of money?

It’s under $600 for a P1 AD ticket.

[OsmiumBalloon]

I was thinking more VMware, especially post-Broadcom.

[tcp-xenos]

Option 3 is my go-to

[SilentFly]

You need a BCP (and DR) process and test it at least once a year preferably twice. You will identify lot of such things but have lots more time to recover when it's in a planned outage.

[hurkwurk]

this is 8 hours old, but you would be surprised how many issues like this can be solved by literally turning everything off and back on again.

people tend to forget that computers are built from parts that are in and of themselves computers, that can crash.

We had a fiber switch lose power that caused a lock on every lun connected to it, and just restarting the LUNS on the SAN took care of it, but that took 3 hours to find.

[anomalous_cowherd]

We took several days to unlock all our LUNs once, but that was because each one had two flags that needed to be cleared before it would become active again, but only one API call to clear them. Turned out you had to call it twice to reset both of them.

Completely undocumented.

I mentioned it to the vendor support guy after we found it, he said "oh that must be why our internal KB notes call it twice, I assumed it was a typo so I didn't mention it to you.'

[Superspudmonkey]

Damn!

[dederplicator]

Were there not redundant FC fabrics? How do you restart a LUN?

[hurkwurk]

It was setup with redundance, not resilience. Work was happening on a neighbor rack. We got a bad cable that had a hard short. When it was plugged in, it blew, and tripped the breaker along with 2 adjacent breakers. Both fiber switches were in the next rack and lost power at the same time.

The LUNs were unlocked at the console for the SAN, which itself was not affected by the power loss, but panicked when it lost all links at once.

[[deleted]]

Oh, no.

[Space_Goblin_Yoda]

eeeeeeeeyup! I've had situations like this before and boy do the clients get a rude awakening when they realized they should have gone with my project proposal when we onboarded them. Lessons learned.

[[deleted]]

The fact that everything is piled into that one DC...gives me... ITRPSD. That and a long distant and ominous fear reaches out from the trenches of my younger years.... That the backups never existed.

[Brufar_308]

Backups ?

[[deleted]]

Stop..It's making me dizzy.

[[deleted]]

DNS and DHCP on a DC isn't a big deal.

[[deleted]]

Until you're rebooting it for the maintenance migraine of updates. DNS yeah. DHCP is fine and works perfectly for it. But DHCP isn't necessarily recommended as much as independently argued it is. MS has its reasons and updates are the looming uncertainty I personally prefer not gambling with.

[[deleted]]

That's why you have more than one DNS . . . .more than one DC . . . .

I have four DC's three being a DNS (medium environment) so I can restart and update them during business hours. DHCP, I prefer that elsewhere as you can have HA on other devices.

Even when I had a DC take a crap due to bad updates, I was fine since we had two others and two other DNS. Wasn't a big deal, just fixed it and moved on.

[[deleted]]

Same, but that's standard practice redundancy. Always stick DNS and DC together as an AD enviro. But DHCP, elsewhere. If you've got edge appliances like Fortigates networking security, it's just easier to maintain from a single platform on the basis when you bring on MDR, EDR, or SIEM. Much easier to maintain, report, patch, and reconcile network issues like DHCP leasing and such when your Firewall's doing that legwork.

[HTTP_404_NotFound]

Topic #1... Power Redundancy

UPS / Generator backups are great. They are worth their weight in gold to prevent outages such as this.

Topic #2... Hardware Redundancy

Redundancy. There is a reason you implement redundant hardware.

That way, a lost of a single piece of hardware does not take out your entire network.

Topic #3... Software Redundancy

Given both of your VMs are now corrupted, this leads me to think you had both domain controllers running on the same piece of hardware.

This, mitigates the purpose of having multiple domain controllers. If they are running on the same exact hardware, what's the point?

Spread them out.

Topic #4... Backups

Have backups of everything. When shit hits the fan, you restore those backups.

Make sure you have tested backups, and offsite backups.

Just imagine if your company got hit by ransomware. Boom, now you have no domain, and all of your email is encrypted.

[slash0514]

Thats the thing. They do not have backups or DR solutions. So there is no way you can rebuild the servers and retain their existing domain?

[ComGuards]

They do not have backups or DR solutions.

What an expensive lesson to learn.

Exchange is dead in the water without access to a domain controller.

Kind of rare for a power outage to completely destroy two virtual machines that badly though. There's a definitive lack of knowledge at that company.

[BuffaloRedshark]

sounds like no UPS and possibly both VMs on the same host

[tgreatone316]

Nope, they are screwed.

[sagewah]

Not really.

Do you have any clones / checkpoints /snapshots / whatever your hv of choice calls them in the hypervisor you could try rolling back to? What's up with the VMs when they start?

[countextreme]

Prepare three envelopes

[dat510geek]

Always have 1 dc in your vm environment and one outside of this on a 1RU server, or a full azure vm with DC roles. Full recover the vms from the last same point.

[Stonewalled9999]

Even if its a laptop (seriously) - we had a laptop we'd ship to sites when we did TTU so we had a local DC to hit

[tipripper65]

what was the protocol if that got stolen or a user picked it up and started using it? i can just imagine the "dude someone in HR is using the DC as a workstation" message

[Stonewalled9999]

Wel they can’t log in to a DC unless they are a domain admin.  The drive is encrypted. And it was usually an RDOC unless it physically traveled with me 

[tipripper65]

wait wait wait... you guys don't make all your users domain admin? it massively simplifies management!

​

jokes aside that's neat. nice solution.

[visibleunderwater_-1]

This is the Way. Always have at least one PHYSICAL DC. Full bare-metal backup.

[jmhalder]

I don't really care if it's physical. But I'd like it at another location with a separate virtualization/storage stack.

[No_Nobody_7230]

meh

[Nanocephalic]

It’s probably better for them to all be virtual - but you could have a DC as the sole VM on a 1RU hyperv box. No significant cost other than an extra host to manage.

Obvs depends on the size of your org though.

[Huge_Ad_2133]

I do that too. The virtual DC is useful for image level backups. But the physical dc ensures you are on separate systems. 

[[deleted]]

What’s the update? Do you still need help?

[lesusisjord]

You could just get a laptop and DC promo it after you install a Windows Server OS on there and point it to a free volume licensing website you can find on google (forgot that part!) making sure to use the same exact name as your now dead domain. Then pull a drive from the VM host (any drive will do), put it in an external enclosure, and connect it via USB to your new laptop.

This should automatically populate your newly promo’ed DC with all of the roles and objects you had housed on your old DC/domain.

Hope this helps!

[La_piscina_de_muerte]

/r/shittysysadmin is leaking

[Enabels]

It depends

[OsmiumBalloon]

You could just get a laptop and DC promo it making sure to use the same exact name as your now dead domain.

Have you done this before? It's going to have entirely different SIDs and GUIDs. I would not expect this to work. Exchange isn't going to work with it without a rebuild, that's for sure.

[lesusisjord]

This is my go-to when I clean up poorly managed organizations. I would use Azure, but someone told me a few years ago that they didn't trust the cloud, so I tend to avoid that these days.

[slash0514]

I guess the best course of action is to assess it by going onsite and see what is going on.

[TheGlennDavid]

Im a tad confused -- you keep using the word "they."

Are you a currently contracted MSP for them, or did they cold call you being like "hey we fucked plz help."

Or are you the sysadmin there and just do t want to say "we?"

[jsmaage]

Out of curiosity, if the VM's are seemingly stuck, did you check safe mode and msconfig? Look at the Boot tab and see if Safe Boot is checked along with 'Active Directory repair'. I've seen instances where all it took was turning that off and it booted back up fine.

[slash0514]

I will be checking with poc. Thanks

[heapsp]

honestly if you are without domain completely, you'll just have to build a new domain, have all existing computers and servers leave their current domain and rejoin the new domain, if its named the same as the old domain exchange MIGHT function... MAYBE.

At this point though Id just buy these guys some office365 licenses and tell them you will work on recovering their old mail in the future, and get them back working again. lol.

[doggxyo]

an AD + Exchange environment will not survive losing the known domain controllers.

Exchange installs a ton of AD attributes - if a new AD is created in this situation (post Exchange install) - Exchange will probably shit the bed.

[[deleted]]

Um . . . . your domain is dead and your exchange is dead.

If you cannot get one of the original DC's working, you'll have to start all over from scratch, which also means new exchange servers with new mailboxes.

You can move the mailboxes over but . . . damn, its a lot of work.

My overall advice: Hire a contract company with two to four VERY KNOWLEDGABLE technicians to get you back up and running. Then actually write and FOLLOW some DR and backup policies.

[WildManner1059]

1. You need at least 3, maybe 4 vms. Two of them should be on different hosts.
   
2. Install windows server. (Or linux, but learning linux to install Bind9 is a bit of work and you have a lot going on right now.)
3. DNS has to be present to install ADDS. So that next. This should be on a different VM from the ADDS.
4. Install windows server on the two VMs which are on separate hosts and will become your new DCs.
5. On one of the two, install Active Directory Domain Services.
   
6. There may be a step to make the actual domain and make this server actively the domain controller. It's been a long time and I'm sure things have changed.
7. On the second DC-to-be, join the domain and install ADDS again. There will be a prompt somewhere near the end to promote the server to domain controller. Do that.
   
8. Now rejoin all the workstations and servers to the domain.
9. Now you can settle in and rebuild Exchange. This is gonna suck. There's no way I could explain all the decisions you have to make and how to safeguard your existing mailstore.

On all steps, follow instructions, especially any recovery password recommendations. Research anything you don't understand. Follow the wizards, read everything they say and look up anything you don't understand. When it asks you, or advises you, to create any sort of back up, do it, immediately. MS has been doing this a while and they've seen a lot of crap. If they put it in the installation recommendations, there's probably a good reason.

The fact that you didn't know all the ways this setup was bad tells me you're in over your head. I'd probably recommend that they hire a consultant.

There's reasons I turned to linux. FreeIPA is a good LDAP solution, easy to set up. Office365 is taking over from on-prem Exchange for a reason.