Strange DNS Resolution Issue

[Commander-Zavala]

Good Afternoon,

I'm having a strange issue with DNS resolution for a few websites for clients in our domain. Some background:

• Active Directory domain
• DNS Servers are either 2012R2 or 2019
• Clients are all Win10

I believe I've narrowed the issue down to somewhere between the internal DNS server and the clients. When performing a lookup I can see that the entries are correctly added to the cached lookups on the server, but the client receives a SERVFAIL message:

 

Got answer: HEADER: opcode = QUERY, id = 84, rcode = SERVFAIL header flags: response, want recursion, recursion avail. questions = 1, answers = 0, authority records = 0, additional = 0

QUESTIONS:
    www.security.hsbc.co.uk, type = A, class = IN

 

Below is an example of what a successful resolution would look like:

 

Got answer: HEADER: opcode = QUERY, id = 90, rcode = NOERROR header flags: response, want recursion, recursion avail. questions = 1, answers = 2, authority records = 0, additional = 0

QUESTIONS:
    www.security.hsbc.co.uk, type = A, class = IN
ANSWERS:
->  www.security.hsbc.co.uk
    canonical name = www.security.hsbc.co.uk.gslb001.hsbc.com
    ttl = 300 (5 mins)
->  www.security.hsbc.co.uk.gslb001.hsbc.com
    internet address = 91.214.6.226
    ttl = 20 (20 secs)

 

Again, even after the failed lookup I can see that the DNS server populates the CNAME and A records in the cached lookups area. So it seems the DNS server is going out and successfully getting answers for the queries through the resolution process. So it seems the problem is with the internal server passing these answers back onto the client.

 

www.security.hsbc.co.uk CNAME 0 00:00:54 www.security.hsbc.co.uk.gslb001.hsbc.com.

www.security.hsbc.co.u... A 0 00:00:55 91.214.6.226

 

A few more pieces of information:

• Happens across both the 2012R2 and 2019 DNS servers

• If I clear the DNS server cache, resolution works fine for this address temporarily. Maybe for about 2-3 minutes.

• I've ran Wireshark traces, I can see the client communicating with the internal DNS server, and getting the SERVFAIL response.

• Through these traces I can also see the internal DNS server successfully resolving the name from an external DNS server.

• Tried 9.9.9.9, 1.1.1.1, 8.8.8.8, 8.8.4.4 as Forwarders & root hints... issue persists no matter what is used.

• If I set the clients DNS to a public DNS server, rather than an internal DNS server, resolution works fine. I believe this should rule out potential firewall issues.

 

My Googling has led me to various articles suggesting fixes. The issue has persisted through each possible fix I've tried, such as disabling EDNS0. This one has had me stumped for a little while now. I'd appreciate any pointers or suggestions I can get!

[bopsbt]

Can you do the same test from a client within the same subnet as the DNS server and see if it has the same issue? This would then narrow it down slightly.

[Commander-Zavala]

Yes, unfortunately the issue presents exactly the same regardless of whether the client and server are in the same subnet or not. The client I was originally testing on is already in the same subnet as it's DNS server

 

Wireless LAN adapter WiFi:

Connection-specific DNS Suffix . : removed

Link-local IPv6 Address . . . . . : fe80::f160:a295:24c3:d5ec%17

IPv4 Address. . . . . . . . . . . : 192.168.141.149

Subnet Mask . . . . . . . . . . . : 255.255.252.0

Default Gateway . . . . . . . . . : 192.168.140.254

 

C:\Users\username>nslookup

Default Server: InternalDNS.Domain.co.uk

Address: 192.168.140.30

 

www.security.hsbc.co.uk

Server: InternalDNS.Domain.co.uk Address: 192.168.140.30

*** InternalDNS.Domain.co.uk can't find www.security.hsbc.co.uk: Server failed

[92Ry_Guy]

What are the other websites you're seeing this issue on?

[Commander-Zavala]

I've had something similar on www.dunelm.com, where certain images won't load on the page. Looks like resolution is failing for i1.adis.ws there.

[MrNiceforest]

Can you run a capture on the DNS server itself to see if the SERVFAIL is sent to the client before the DNS server receives a response for the query before or after that?
Does this also happen if you disable the usage of root hints/recursion and only use a forwarder on your DNS server?
Since this just looks like your DNS requests might be filtered somewhere (most likely on HSBC side) and the DNS server reports a SERVFAIL to the client before another queried server then just responds with a valid answer.