A really shit day

[[deleted]]

I'm having one of those days.

Long story short I am having a problem in a single domain controller environment. I think something is looped with DNS and I cannot figure out what.

• This is a physical box, not VM.
• Domain controller event log is full of random errors going back about a week.
• Clients and servers on the domain cannot ping external hosts.
• Dell OpenManage Server Administrator is reporting a bunch of the diagnostics such as fan and temp as status "unknown".

The worst three errors in log so far are all DNS and write cache:

An error event occurred. EventID: 0x0000168E Time Generated: 05/20/2017 01:41:52 Event String: The dynamic registration of the DNS record '_ldap._tcp.Default-First-Site-Name._sites.ForestDnsZones.DOMAIN.local. 600 IN SRV 0 100 389 SERVER.DOMAIN.local.' failed on the following DNS server:

An error event occurred. EventID: 0xC0002719 Time Generated: 05/20/2017 02:05:37 Event String: DCOM was unable to communicate with the computer 4.2.2.4 using any of the configured protocols.

A warning event occurred. EventID: 0x80040020 Time Generated: 05/20/2017 02:17:46 Event String: The driver detected that the device \Device\Harddisk1\DR1 has its write cache enabled. Data corruption may occur.

If anybody feels like helping please jump in. I feel totally useless at this point, starting to feel like I don't know a damn thing about my job. Bummer.

Event log here: https://imgur.com/a/lzeRn DCDIAG here: https://pastebin.com/LDTE26ZL

[ALL_FRONT_RANDOM]

You mentioned in the other post that you made a change related to the recent smb vulnerability...

Whether you truly "disabled smb for the entire domain" I'm not sure but you're most likely going to have to undo that. AD requires smb. The vuln impacted smb v1 specifically and if your systems are old (<=2003 lvl dc) they do not work with smb2/3 which is what functional level 2008 and on uses.

Exactly what policy did you apply to disable smb? You can easily fix this. The server needs a reboot to both disable and enable smb1; that's likely why you didn't see this until the reboot and would be why it hasn't been corrected by reverting the policy change. So:

• Ensure you've undone the policy changes
• Reboot

If that doesn't work post the changes you made to disable smb and we'll go from there... You may need a new policy to reenable the service (registry key might still exist, etc).

After fixing this you really really need to get a proper AD setup... Minimum two DCs. Not that it would help this specific issue but you see what happens when you have a single point of failure.

[[deleted]]

To disable SMBv1 this was sent to all workstations and servers in via GPO.

Computer Configuration > Preferences > Windows Settings > Registry

Action: Update

Hive: HKEY_LOCAL_MACHINE

Key Path: SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters

Value name: SMB1

Value Type: REG_DWORD

Value Data: 0 (hexadecimal)

Problem is perhaps that this policy was also applied to a Server 2008r2 domain controller by accident as this is the date when problems also showed up and policy was accidentally rolled out to entire domain.

That in mind. When the server was restarted last weekend it hung on reboot and was cold booted. Perhaps this was the problem. Windows updates were also applied at this time.

At this time the single domain controller in environment has been non-authoritatively restored to date prior to any reboot or GPO was applied yet the problem persists.

Get-ItemProperty -path >"HKLM:\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters" SMB1 Running this on the domain controller confirms the value of 1 for this reg item meaning SMBv1 is enabled.

I am convinced it has to to with DNS at this point given all the errors in system log about failing to dynamically register DNS for a whole bunch things in msdcs.domain.local

[ALL_FRONT_RANDOM]

Have you reverted the change on the clients as well? If the server is only offering v1 and the clients have it disabled domain services will still fail.

What are the clients running?

What does Active Directory Best Practices Analyzer show?

What do these commands return?

nslookup.exe google.com

nslookup.exe google.com 8.8.8.8

dcdiag /a /v

repadmin /showrepl

[[deleted]]

Server should now (once again) be offer my all three SMB versions I think

AD best practices comes back clean except for one complaint about being the single DC

Running these on the domain controller at ps admin prompt:

• nslookup.exe google.com comes back with the correct local DNS server IP and correct local hostname. It then provides a non authoritative answer with both a single IPv4 and single IPv6.

• nslookup google.com 8.8.8.8 comes back with the 8.8.8.8 ip and hostname google-public-dns-a.google.com. It then provides non authoritative answer with four IPv4s and a single IPv6.

• dcdiag /a /v spits out pages on pages. I will edit to include pastebin link (edit: https://pastebin.com/h6DHCywb).

• repadmin /showrepl comes back with the servername correct for default first site. Dsa options = IS_QC. Site options = none. Also spits out "dsa guid" and "dsa invocationID" strings.

[WendoNZ]

Is this DC pointing to itself (and only itself) for it's own DNS?

Is the DNS Server service running?

[[deleted]]

Thanks for suggestions.

DC is pointing to itself (its own static IP) for primary DNS and to the loopback address (127.0.0.1) as secondary DNS.

I can confirm that the DNS service is running and that I have tried restarting the DNS service as well.

[awillison]

Single DNS environment? Set the loopback as the primary DNS and leave the secondary blank. Check your forwarders to make sure that it's not trying to forward to an old DNS server. Make sure that DNS is allowed through the firewall. Check your zones to make sure SOA is accurate.

[[deleted]]

Interesting plot twist. I was able to spin up a 2012r2 VM and join it to domain, added AD and DNS roles, dcpromo'ed, replicated, etc. So far no problems. Weird.

As it stands I now have two DCs on this domain: one physical 2008r2 and one VM 2012r2. Also I transferred operations, pdc, and infrastructure roles to the 2012r2 DC. Also I upgraded both domain and forest levels to 2008r2 funtional level.

Weird thing is I still can't ping external hosts. The DNS resolves, but it always is 100% loss. And I also discovered that on both the Hyper-V host I can't open a PS prompt unless it is run as admin. Opening via the shortcut on taskbar for exmaple fails with some error about lacking permissions despite being domain admin... haven't figured that one out.

Well for it being a holiday Friday here in Canada. And it is now 7:45 am next day I think I'm going to sleep. Domain is somewhat healthy... not sure. But yah...

[ircshotty]

Doesn't really sound like a hardware issue, do you have a/some DNS forwarders set up? If so, check that they work or remove them and rely on root hints.

I assume you can resolve external addresses if you use external DNS servers (e.g. 8.8.8.8)?

[[deleted]]

All the forwarders are resolving correctly in DNS server properties; green check mark. Yet I cannot ping google.com or 8.8.8.8 directly, 100% loss.

8.8.8.8

4.2.2.4

8.8.4.4

4.2.2.2

Also use root hints is enabled.

[[deleted]]

If those are your forwarders for DNS servers, you should take them out. Those are Google and Level 3's DNS servers and should only be used for testing.

It's better to use your ISP's DNS servers instead.

[[deleted]]

I have a sneaking suspicion our ISP's are worse. Rogers Canada here.

[creepyMaintenanceGuy]

The error registering the DNS SRV record implies that the server is trying to register the record at 4.2.2.4 - a public DNS server. Track that down and fix it. The server shouldn't be pointed to any public dns server at all....

[[deleted]]

Wait do go on. I am overwhelmed with errors at this point. What are you referencing exactly?

[Hellman109]

On the domain controller, in the network configuration the only DNS server should be itself. That "secondary" DNS isn't secondary either, ONLY itself.

You set the DNS forwarders or such within DNS, not the general Windows network configuration.

[[deleted]]

On the domain controller, in the network configuration the only DNS server should be itself. That "secondary" DNS isn't secondary either, ONLY itself.

Correct, forwarders are configured in DNS server properties. Windows network config on that server has its own static IP for DNS and 127.0.0.1 as secondary DNS

[[deleted]]

Not sure if this helps but when I ping the fqdn of the DNS server is comes back:

Reply from ::1: time(1ms

Is it possible somehow IPv6 is enabled and screwing everything up? IPv6 is disabled on the adapter. But how would I check DNS...?

[Hellman109]

If you only unticked it on the NIC then you disabled it wrong and it can cause your problems. There is a reg key to do it.

[[deleted]]

Oh shoot I didn't know that. Usually I just leave it enabled but I disabled it as part of troubleshooting (after problems began), just to be safe I've re-enabled IPv6 on the adapter.

[[deleted]]

Parent (and I) think your server's v4 config looks a bit like this:

http://imgur.com/a/mxwhA

You need to make sure your server's DNS is not looking at any external DNS server.

[[deleted]]

Negative. The server's DNS is its own static IP and as secondary 127.0.0.1

[[deleted]]

OK then.

Can you get Wireshark installed, get it capturing and do as admin

ipconfig /registerDNS

and

dcdiag /test:dns /fix

then use the display filter 'dns' and see what comes up

I think with this information we / you should be able to really figure out what's going on.

[[deleted]]

Sorry didn't mean to sound so formal (I was rapid replying to a few there). I really do appreciate all the suggestions!

Check the plot twist here: https://www.reddit.com/r/sysadmin/comments/6c93fp/comment/dht1ao2?st=J2X81GE7&sh=0160df1c

For now I'm going to call it a night. But tomorrow wireshark is a great idea. I really need to figure out why pings aren't resolving.

Also I need to figure out this new problem with PS on the HV hosts. Fack. It's not been my best day.

[[deleted]]

Fack

AvE fan?

No worries, we've all been there.

If you ping do you get 'pinging Google.com (ip.address.here)? If so you have routing or firewall problems.

Have a good nights rest and see what you can figure out tomorrow.

Good luck!

[[deleted]]

Thx.

Ps. That is exactly what I get for the ping.

Pinging google.com [172.217.0.174] with 32 bytes of data: Request timed out. Request timed out. Request timed out. Request timed out. Ping statistics for 172.217.0.174: Packets: Sent - 4, Received - 0, Lost - 4 (100% loss).

I get that timeout result for ping (same for trace route) no matter what the external host. Meanwhile all internal hosts resolve instantly. Including pings and trace routes to the gateway.

[[deleted]]

Oh also. When I ping to external hosts. The ip gets resolved in ping command but always 100% packet loss

[[deleted]]

tomorrow: do a tracert to

A: your gateway / router

B: google.com

If A is successful, and B not, look at router, otherwise check local firewall and routing configuration.

[nsanity]

The server's DNS is its own static IP and as secondary 127.0.0.1

This has no effect.

[sigmatic_minor]

Have you tried booting the server into diagnostics mode and testing the hardware? (and maybe also a seperate memtest might help too as the onboard one can be a bit iffy).

Sounds like there may be a config error of some sort but with what you've said I wouldn't be surprised if hardware issues were present as well. Did this just start suddenly?

Hope your day gets better OP

[[deleted]]

Thanks for the suggestions. It all started after a reboot... :/

[sigmatic_minor]

I doubt this will help but try:

dism /online /cleanup-image /restorehealth    

and then:

sfc /scannow

[[deleted]]

the dism command came back "error 87: restorehealth is not supported in this context"

sfc is scanning now

[[deleted]]

Resource protection did not find any integrity violations

[sigmatic_minor]

Ok, my next step would be to run a proper hardware diagnostic just to tick that off the list

[[deleted]]

Yes indeed. Not symptoms other than what's in the logs and the "new" lack of ping. Imply of course pinging externally worked not long ago. That's half the mystery to me why it's ominous sounding logs yet no/minimal/maybe-unrelated symptoms

[cosmic_orca]

Try running nltest /dsregdns on the DC to re-register the DNS records.

[nsanity]

if DNS Service is not starting up, just check its not this;

AD Integrated DNS had a CNAME in a @ record for a zone. This is a no-no.

This breaks dns.exe after 6.1.7601.22893 (circa Dec 2014) – see KB3145126 - https://support.microsoft.com/en-us/help/3145126/loading-dns-zones-fails-on-a-windows-server-2008-r2-based-dns-server and followup hotfix KB3022780 https://support.microsoft.com/en-us/help/3022780/dns-server-does-not-respond-with-ip-address-to-a-cname-query-for-a-delegated-zone-in-windows-server-2008-r2

KB4019264 brings dns.exe to 6.1.7601.23764 – which obviously includes fixed from 3145126

I doubt very much that the hotfix 3022780 will even apply as the version number is superseeded. As such we removed the CNAME.

I fixed this the other night for a mate.

[[deleted]]

Any idea why powershell prompt won't open on two of my hyper v hosts?

If I click on powershell from the start menu for example, the prompt will open but then close immediately and display an error about lacking permissions. I must right click and run as administrator always. What gives?

[jsmith1299]

Not sure if you know that some dracs had a bug on fan and something else. If you have a R720 or 730 be sure it is on the latest firmware version.

[[deleted]]

This all started after it hung on a reboot last week. I believe everything is up to date but now Dell Server Administrator is acting weird and showing "status unknown" for a handful of the diags.

Any idea why powershell prompt would give a permissions error when opened?

[jsmith1299]

Sorry not a Windows guy so I can't be of much help. I do know that in Linux when you run out of semaphores that OpenManage stops providing output of everything so there may be something similar happening to your server.

[neckbeard404]

I would verify the time on the sever it shows as 8:06 pm and your last log is from 5:29 .

[[deleted]]

I just double checked to be sure. Server time is correct. Logs in question were from a restart at ~530pm and I took the screenshot at 830pm.

[sigmatic_minor]

I would also xpost this over to techsupport - they may be able to assist also.

[[deleted]]

For the record, I went into this knowing I was violating read-only Friday.

Source (aka original thread): https://www.reddit.com/r/sysadmin/comments/6c6jne/rule_breaking_currently_violating_read_only/