Tag: DNS

Exchange 2013 DNS Oddity

Not that anyone hosts their own Exchange server anymore … but we had a pretty strange issue pop up. Exchange has been, for a dozen years, configured to use the system DNS servers. The system can still use DNS just fine … but the Exchange transport failed to query DNS and just queued messages.

PS C:\scripts> Get-Queue -Identity "EXCHANGE01\3" | Format-List *

DeliveryType : SmtpDeliveryToMailbox
NextHopDomain : mailbox database 1440585757
TlsDomain :
NextHopConnector : 1cdb1e55-a129-46bc-84ef-2ddae27b808c
Status : Retry
MessageCount : 7
LastError : 451 4.4.0 DNS query failed. The error was: DNS query failed with error ErrorRetry
RetryCount : 2
LastRetryTime : 1/4/2025 12:20:04 AM
NextRetryTime : 1/4/2025 12:25:04 AM
DeferredMessageCount : 0
LockedMessageCount : 0
MessageCountsPerPriority : {0, 0, 0, 0}
DeferredMessageCountsPerPriority : {0, 7, 0, 0}
RiskLevel : Normal
OutboundIPPool : 0
NextHopCategory : Internal
IncomingRate : 0
OutgoingRate : 0
Velocity : 0
QueueIdentity : EXCHANGE01\3
PriorityDescriptions : {High, Normal, Low, None}
Identity : EXCHANGE01\3
IsValid : True
ObjectState : New

Yup, still configured to use the SYSTEM’s DNS:

PS C:\scripts> Get-TransportService | Select-Object Name, *DNS*

Name : EXCHANGE01
ExternalDNSAdapterEnabled : True
ExternalDNSAdapterGuid : 2fdebb30-c710-49c9-89fb-61455aa09f62
ExternalDNSProtocolOption : Any
ExternalDNSServers : {}
InternalDNSAdapterEnabled : True
InternalDNSAdapterGuid : 2fdebb30-c710-49c9-89fb-61455aa09f62
InternalDNSProtocolOption : Any
InternalDNSServers : {}
DnsLogMaxAge : 7.00:00:00
DnsLogMaxDirectorySize : 200 MB (209,715,200 bytes)
DnsLogMaxFileSize : 10 MB (10,485,760 bytes)
DnsLogPath :
DnsLogEnabled : True

 

I had to hard-code the DNS servers to the transport and restart the service:

PS C:\scripts> Set-TransportService EXCHANGE01 -InternalDNSServers 10.5.5.85,10.5.5.55,10.5.5.1
PS C:\scripts> Set-TransportService EXCHANGE01 -ExternalDNSServers 10.5.5.85,10.5.5.55,10.5.5.1

PS C:\scripts> Restart-Service MSExchangeTransport
WARNING: Waiting for service 'Microsoft Exchange Transport (MSExchangeTransport)' to stop...
WARNING: Waiting for service 'Microsoft Exchange Transport (MSExchangeTransport)' to start...

PS C:\scripts> Get-TransportService | Select-Object Name, InternalDNSServers, ExternalDNSServers

Name InternalDNSServers ExternalDNSServers
---- ------------------ ------------------
EXCHANGE01 {10.5.5.1, 10.5.5.55, 10.5.5.85} {10.5.5.85, 10.5.5.55, 10.5.5.1}

 

Viola, messages started popping into my mailbox.

ISC Bind 9.18 and Windows DNS

After upgrading all of our Linux hosts to Fedora 39, we are running ISC bind 9.18.21 … and it seems the ISC folks are finally done with Microsoft’s “kinda sorta RFC compliance”. Instead of just working around Windows DNS servers having some quirks … they now fail to AXFR the domain.

Fortunately, you can tell bind to stop doing edns ‘stuff‘ by adding a server{} section to named.conf — this gives the server some instructions on how to communicate with the listed server. When bind is no longer trying to do edns “stuff”, Windows doesn’t have an opportunity to provide a bad response, so the AXFR doesn’t fail.

Web Proxy Auto Discovery (WPAD) DNS Failure

I wanted to set up automatic proxy discovery on our home network — but it just didn’t work. The website is there, it looks fine … but it doesn’t work. Turns out Microsoft introduced some security idea in Windows 2008 that prevents Windows DNS servers from serving specific names. They “banned” Web Proxy Auto Discovery (WPAD) and Intra-site Automatic Tunnel Addressing Protocol (ISATAP). Even if you’ve got a valid wpad.example.com host recorded in your domain, Windows DNS server says “Nope, no such thing!”. I guess I can appreciate the logic — some malicious actor can hijack all of your connections by tunnelling or proxying your traffic. But … doesn’t the fact I bothered to manually create a hostname kind of clue you into the fact I am trying to do this?!?

I gave up and added the proxy config to my group policy — a few computers, then, needed to be manually configured. It worked. Looking in the event log for a completely different problem, I saw the following entry:

Event ID 6268

The global query block list is a feature that prevents attacks on your  network by blocking DNS queries for specific host names. This feature has caused the DNS server to fail a query with error code NAME ERROR for wpad.example.com. even though data for this DNS name exists in the DNS database. Other queries in all locally authoritative zones for other names
that begin with labels in the block list will also fail, but no event will be logged when further queries are blocked until the DNS server service on this computer is restarted. See product documentation for information about this feature and instructions on how to configure it.

The oddest bit is that this appears to be a substring ‘starts with’ query — like wpadlet or wpadding would also fail? A quick search produced documentation on this Global Query Blocklist … and two quick ways to resolve the issue.

(1) Change the block list to contain only the services you don’t want to use. I don’t use ISATAP, so blocking isatap* hostnames isn’t problematic:

dnscmd /config /globalqueryblocklist isatap

View the current blocklist with:

dnscmd /info /globalqueryblocklist

– Or –

(2) Disable the block list — more risk, but it avoids having to figure this all out again in a few years when a hostname starting with isatap doesn’t work for no reason!

dnscmd /config /enableglobalqueryblocklist 0

 

Linux: Disabling Wild Local DNS Server Thing (i.e. systemd-resolved)

I am certain there is some way to configure systemd-resolved to actually use internal DNS servers so you can resolve your local hostnames. But nothing I’ve tried have worked, and I don’t actually need this wild local DNS thing.

Here’s the problem — systemd-resolved creates an /etc/resolv.conf file that uses a localhost address as the nameserver — and that may very well forward requests out to Internet DNS servers. Which don’t have any clue about your internal DNS zones — thus you can no longer resolve local hostnames. Whenever I see 127.0.0.53 in /etc/resolv.conf, I know systemd-resolved is at work.

[lisa@linux ~]# cat /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search example.com

To disable this local name resolution, stop and disable systemd-resolved, unlink the /etc/resolv.conf file it created, and restart NetworkManager

[lisa@linux ~]# systemctl stop systemd-resolved.service
[lisa@linux ~]# systemctl disable systemd-resolved.service
[lisa@linux ~]# unlink /etc/resolv.conf
[lisa@linux ~]# systemctl restart NetworkManager
Removed /etc/systemd/system/multi-user.target.wants/systemd-resolved.service.
Removed /etc/systemd/system/dbus-org.freedesktop.resolve1.service.

Voila, /etc/resolv.conf is now populated with reasonable internal DNS servers, and you can resolve local hostnames.

[lisa@linux ~]# cat /etc/resolv.conf
# Generated by NetworkManager
search example.com
nameserver 10.1.2.33
nameserver 10.1.2.66

Maintaining an /etc/hosts record

I encountered an oddity at work — there’s a server on an internally located public IP space. Because it’s public space, it is not allowed to communicate with the internal interface of some of our security group’s servers. It has to use their public interface (not technically, just a policy on which they will not budge). I cannot just use a DNS server that resolves the public copy of our zone because then we’d lose access to everything else, so we are stuck making an /etc/hosts entry. Except this thing changes IPs fairly regularly (hey, we’re moving from AWS to Azure; hey, let’s try CloudFlare; nope, that is expensive so change it back) and the service it provides is application authentication so not something you want randomly falling over every couple of months.

So I’ve come up with a quick script to maintain the /etc/hosts record for the endpoint. 

# requires: dnspython, subprocess

import dns.resolver
import subprocess

strHostToCheck = 'hostname.example.com' # PingID endpoint for authentication
strDNSServer = "8.8.8.8"         # Google's public DNS server
listStrIPs = []

# Get current assignement from hosts file
listCurrentAssignment = [ line for line in open('/etc/hosts') if strHostToCheck in line]

if len(listCurrentAssignment) >= 1:
        strCurrentAssignment = listCurrentAssignment[0].split("\t")[0]

        # Get actual assignment from DNS
        objResolver = dns.resolver.Resolver()
        objResolver.nameservers = [strDNSServer]
        objHostResolution = objResolver.query(strHostToCheck)

        for objARecord in objHostResolution:
                listStrIPs.append(objARecord.to_text())

        if len(listStrIPs) >= 1:
                # Fix /etc/hosts if the assignment there doesn't match DNS
                if strCurrentAssignment in listStrIPs:
                        print(f"Nothing to do -- hosts file record {strCurrentAssignment} is in {listStrIPs}")
                else:
                        print(f"I do not find {strCurrentAssignment} here, so now fix it!")
                        subprocess.call([f"sed -i -e 's/{strCurrentAssignment}\t{strHostToCheck}/{listStrIPs[0]}\t{strHostToCheck}/g' /etc/hosts"], shell=True)
        else:
                print("No resolution from DNS ... that's not great")
else:
        print("No assignment found in /etc/hosts ... that's not great either")

ISC Bind – Converting Secondary Zone to Primary

Our power went out on Monday and, unfortunately, the SSD on the server with all of our VMs got corrupted. The main server has ISC Bind configured to host all of our internal DNS zones as secondaries … but, a day after the primary DNS server went down, those copies fell over. Luckily, you can convert a secondary zone to primary. The problem is that the cached copy of the zone was … funky binary stuff.

Luckily there’s an executable to convert this into a text zone file — named-compilezone

-f raw -F text -o output_file_name zone_name input_file_name

So, to covert my rushworth.us zone:

named-compilezone -f raw -F text -o rushworth.us.db rushworth.us rushworth.us.db.bin

Then, in the named.conf file, change the zone type to “master” and remark out the line indicating which the masters are. Change the “files” line to the newly created file. If you haven’t already done so, add “allow-query {any; };” so clients can actually query the zone.

Porkbun DDNS API

I’ve been working on a script that updates our host names in Porkbun, but the script had a problem with the example.com type A records. Updating host.example.com worked fine, but example.com became example.com.example.com

Now, in a Bind zone, you just fully qualify the record by post-pending the implied root dot (i.e. instead of “example.com”, you use “example.com.”, but Porkbun didn’t understand a fully qualified record. You cannot say the name is null (or “”). You cannot say the name is “example.com” or “example.com.”

In what I hope is my final iteration of the script, I now identify cases where the name matches the zone and don’t include the name parameter in the JSON data. Otherwise I include the ‘name’ as the short hostname (i.e. the fully qualified hostname minus the zone name). This appears to be working properly, so (fingers crossed, knock on wood, and all that) our ‘stuff’ won’t go offline next time our IP address changes.

It’s Not A DNS Problem

I used to work at a company where everything was called an Exchange problem — not that Exchange 2000 didn’t have it’s share of problems (store.exe silent exit failures? Yes, that’s absolutely an Exchange problem) … but the majority of the time, the site had lost their connectivity back to the corporate data center. Or, when I’d see the network guys sprinting down the hallway as the first calls started to come in … the corporate data center had some sort of meltdown.

I’m reminded of this as I see people calling the Facebook outage a “DNS problem”. Facebook’s networks dropped out of BGP routing. That means there’s no route to their DNS server, so you get a resolution failure. It doesn’t mean there’s a DNS problem. Any more than it means there’s an IP or power problem — I’m sure it’s all working as designed and either someone screwed up a config change or someone didn’t screw up and was trying to drop them off the Internet.

Saw much the same thing back when Egypt dropped off of the Internet back in 2011 — their routes were withdrawn from the routing tables. That’s an initiated process — maybe accidental, but it’s not the same as a bunch of devices losing power or a huge fiber cut.

And, when there’s no route you can use to get there … if DNS, web servers, databases, etc are working or not becomes moot.

Testing A New Web Server Without DNS Changes

When migrating to a new server, it’s good to validate site functionality before redirecting users to the new host. i.e. I have anya.rushworth.us set up in the httpd config on both server1 and server2. DNS currently points traffic to server1, but I need to test the site on server2.

Approach #1 – With administrative access to the host

Edit your hosts file – open an administrative command prompt

Edit %SYSTEMROOT%\system32\drivers\etc\hosts and add lines with the IP address WHITESPACE and the hostname(s). E.G.
127.0.0.1 lisatest lisatest.rushworth.us lisatest2 lisatest2.rushworth.us
10.1.2.3 otherhost otherhost.rushworth.us
10.2.3.4 anya anya.rushworth.us

Clear your DNS cache (ipconfig /flushdns) and navigate to the URL. You’ll be directed the IP address from your hosts file instead of the DNS registered address.

Approach #2 – No admin access

Install ModHeader in your Chrome browser and click the extension to modify the headers or install ModHeader in your Firefox browser. Click on the extension icon to set a header value.

Add a “Host” header with the value of the virtual host name you need to test

Navigate to the hostname of the new server – https://server2.rushworth.us – but the web server will receive the Host header you configured in ModHeader and serve the web site based on that host header.

 

Response Policy Zone (RPZ)

Years ago, Paul Vixie developed a component of the BIND DNS server that allowed server owners to easily override specific hostnames. We had done something similar for particularly bad hostnames — if your workstations use your DNS servers, you just have to declare yourself the name server for a domain that has the same name as the hostname you want to block (i.e. I become the NS record for forbidden.google.com and my clients are able to resolve all other records within the google.com zone, but when they resolve forbidden.google.com … they get whatever I provide). I usually did this to route traffic over a B2B VPN – provided the private IP address instead of the public IP provided by the domain owner’s name servers. But for a few really bad malware variants, I overrode their hostname. Problem was the technique wasn’t exactly easy. Every single host required a new DNS zone be created, configured on your DNS servers, and (at least in BIND) the service restarted.

Response Policy Zone was pushed as a functionality that would allow service providers (ISPs). That’s not a use case I forsee (it’s a lot of manual work), but it has become an important component of our company’s network security. Hosting an RPZ domain allows us to easily add new overrides for B2B VPN connected hosts. But it also means we can override hostnames that appear in phishing e-mail campaigns, malware hosts, infected web sites … basically anything we don’t want employees accessing.

Stopping clients from accessing infected sites is a great thing; but for hostnames that are indicative of a compromised box (i.e. there’s a difference between an employee clicking on a link within their e-mail that links them to a specific host and someone having malware on their box that automatically contacts a specific host), we set the IP address for the hostname to a honeypot.

The honeypot is bound to all unused ports on the host (there aren’t a lot of used ports on it), logs all contact to a database, then basically hangs the connection. We have a scheduled job that looks at the contact log and opens a ticket to the desktop support team to investigate the compromised host.