Mike Howells's Blog

Tag: dns

  • The Curious Case of aa.ns.charter.com

    The Curious Case of aa.ns.charter.com

    Or: how a stray entry in a Pi-hole log led me to a seven-year-old bug in Charter’s authoritative DNS.

    It started with a line in a log.

    I run Pi-hole on a couple of Raspberry Pis on my home network. It blocks ads and tracking and a long list of telemetry endpoints I’d rather not have my devices phoning home to. Most days I don’t think about it. But occasionally I pull up the query log just to see what’s flowing through, and I scan for anything that looks weird.

    What caught my eye was this:

    aa.ns.charter.com. Hourly. Blocked. From one of my domain controllers.

    Charter is Spectrum, my ISP. The hostname looks like one of their nameservers. The query was coming from inside the house, so to speak. A Windows Server running Active Directory was asking my Pi-hole to resolve this name, on the hour, every hour, and Pi-hole was flagging it as blocked.

    There are a lot of reasons something might end up in a Pi-hole log. Most of them are boring. This one looked boring at first. But I poked at it for a few minutes, and what I thought would be a five-minute mystery turned into something much more interesting.

    This is what I found.

    The setup

    Let me sketch the environment, because some details matter and most don’t.

    I run three Windows Server 2025 domain controllers on my home lab: SKYE, BOYD, and EMMA. They’re all on a flat 192.168.2.0/24 subnet. They handle Active Directory and internal DNS for a domain called howells.lan.

    Outbound DNS from those DCs flows through two Raspberry Pi-hole instances, Pi3 and Pi4. The DCs forward to the Pis; the Pis run dnscrypt-proxy upstream, which uses encrypted DNS to Cloudflare, Quad9, and NextDNS. Pi-hole blocks anything that’s on one of its 33 adlists, and dnscrypt-proxy enforces strict DNSSEC validation on everything that gets through.

    The whole pipeline is designed to be paranoid in a particular way: every external lookup is encrypted, validated, and filtered, with redundancy at every layer.

    The query I noticed was hitting Pi3, logged against SKYE as the client (192.168.2.11), asking for an A record for aa.ns.charter.com. Pi-hole was returning 0.0.0.0 and marking it blocked.

    My first thought was: which of the 33 adlists is catching a Charter nameserver hostname? That seemed odd. Adlists usually target ad networks and trackers, not ISP infrastructure.

    The first wrong turn

    Pi-hole has a built-in tool for exactly this question. It’s called “Find Domains in Lists.” You type in a hostname and it tells you, definitively, which of your installed blocklists contain it.

    I typed in aa.ns.charter.com.

    Found 0 domains partially matching 'aa.ns.charter.com'.
    Found 0 lists partially matching 'aa.ns.charter.com'.
    Number of results per type:
    - 0 exact domain matches
    - 0 regex domain matches
    - 0 allowlist (antigravity) matches
    - 0 blocklist (gravity) matches

    Zero.

    So whatever was happening, it wasn’t a blocklist hit. Pi-hole was flagging the query as blocked, but no list of mine was telling it to. That’s interesting. Pi-hole doesn’t just block things on a whim. Something else was producing the “blocked” status.

    I checked the response time in the query log. 69 microseconds for the A record. That’s not a real upstream lookup, that’s a cached or synthesized response. Whatever Pi-hole was doing, it was returning an answer almost instantly.

    I looked at what the upstream resolvers actually said. From Pi3, I ran a direct dig against the dnscrypt-proxy upstream, and then bypassed it and asked Cloudflare and Quad9 directly:

    $ dig aa.ns.charter.com A +short
    0.0.0.0

    All three resolvers, my dnscrypt chain, Cloudflare’s 1.1.1.1, and Quad9’s 9.9.9.9, returned the same thing. aa.ns.charter.com resolves to 0.0.0.0.

    That changed the picture entirely. Pi-hole wasn’t blocking the query in the sense of consulting a list and refusing it. Pi-hole was receiving the authoritative answer 0.0.0.0 from upstream, recognizing it as a null/sinkhole address, and flagging it in the UI as a block. That’s a default Pi-hole v6 behavior: any A record that resolves to 0.0.0.0 (or AAAA to ::) gets treated as a blocked response, regardless of source.

    So the question wasn’t “why is Pi-hole blocking this.” The question was “why does aa.ns.charter.com resolve to 0.0.0.0, and why is my domain controller asking about it once an hour?”

    Two questions, actually. I started with the second one.

    Hunting the process

    If something on SKYE was generating a DNS query every hour, I figured I could find it. Windows is reasonably well-instrumented for this kind of thing if you know where to look.

    I started with active network connections. Get-NetTCPConnection, filtered to anything talking outbound:

    RemoteAddress    RemotePort  OwningProcess
    -------------    ----------  -------------
    204.79.197.203   80          7956
    23.11.32.159     80          7956
    20.1.248.118     443         7956
    20.59.87.225     443         2760

    PID 7956 was making three outbound connections, two of them to Microsoft IP space on port 80. Promising. I looked up the process.

    ProcessName : StartMenuExperienceHost
    Path        : C:\Windows\SystemApps\...
    Description : Windows Start Experience Host

    It was the Start menu.

    I had an active RDP session open on SKYE while I was investigating. The Start menu was sitting on screen, doing what Start menus do, which apparently includes polling Microsoft endpoints every ten seconds for “recommendations” and live tile updates. That’s a real Pi-hole entry I’d been seeing every ten seconds for g.live.com, blocked dutifully. But it wasn’t my hourly Charter query. Wrong process, but I’d at least confirmed I knew how to find the right one.

    The other PID, 2760, was WpnService, the Windows Push Notification Service. That’s expected. It maintains a long-lived TLS connection to Microsoft’s notification infrastructure for toast notifications. Also not my culprit.

    Next I looked at scheduled tasks with run times near the :19 mark, since the Charter queries were landing at :19:32 every hour.

    TaskName                NextRunTime          
    --------                -----------          
    WakeUpAndScanForUpdates 5/19/2026 12:19:23 AM
    Collection              5/18/2026 6:19:56 PM

    Collection. Hourly. Next run at 6:19:56 PM. That was suspiciously close to my Pi-hole pattern.

    The task lived at \Microsoft\Windows\Software Inventory Logging\Collection and ran a command called silcollector.cmd publish as SYSTEM. I’d never heard of SIL (Software Inventory Logging), but a few minutes of reading told me it was a Windows Server feature designed to periodically inventory installed software and licensing data and publish it to a configured target. It was introduced in Server 2012 R2 for datacenter compliance reporting.

    This felt like the answer. A SYSTEM-context task that runs hourly, does some kind of inventory or telemetry, and might reasonably touch the network in the process.

    I checked whether SIL was actually configured to publish anywhere:

    State                 : Stopped
    TimeOfDay             : 1/1/2000 3:00:00 AM
    TargetUri             :
    CertificateThumbprint :

    Nope. SIL was Stopped, no target URI. But the scheduled task that runs silcollector.cmd publish was still enabled and still firing hourly. The cmd file would do its inventory collection regardless of whether anything got published.

    I ran the cmd manually as my admin user, watched Pi-hole’s live query log, and waited.

    6:32:05 PM - silcollector.cmd publish executed

    Within the next 30 seconds, I saw queries for roaming.svc.cloud.microsoft, accounts.google.com, app.ps.five9.com, fonts.googleapis.com, and a handful of other Office and browser telemetry endpoints. All from my active RDP session.

    No aa.ns.charter.com. None.

    SIL wasn’t doing it. The timing was a coincidence.

    I’d been wrong, but I’d at least learned something: whatever was generating the Charter query wasn’t using the standard Windows DNS Client resolver. If it had been, the manual SIL invocation should have surfaced something Charter-related in the immediate aftermath. The process making this hourly query was bypassing the Windows DNS Client API entirely.

    A pattern that didn’t quite fit

    I pulled the full history of aa.ns.charter.com queries from Pi3 going back about 24 hours.

    2026-05-18 19:19:33  skye.howells.lan
    2026-05-18 18:19:33  skye.howells.lan
    2026-05-18 15:19:33  skye.howells.lan
    2026-05-18 14:19:32  skye.howells.lan
    2026-05-18 12:19:32  skye.howells.lan
    2026-05-18 11:19:32  skye.howells.lan
    2026-05-18 10:19:32  skye.howells.lan
    2026-05-18 09:19:27  skye.howells.lan
    2026-05-18 08:28:40  skye.howells.lan
    2026-05-18 05:28:40  skye.howells.lan
    2026-05-18 03:28:40  skye.howells.lan
    ...

    Two things jumped out.

    First, the cadence wasn’t strictly hourly. There were skipped hours. 13:00, 16:00, 17:00, nothing. The pattern was “roughly hourly with occasional misses.” That’s not what you get from a scheduled task with a fixed timer. That’s what you get from a process that does something on a periodic cycle but occasionally skips when conditions don’t match.

    Second, the seconds offset shifted partway through the day. Earlier entries were at :28:39-40. Later ones were at :19:27-33. Something restarted between 8:28 and 9:19 in the morning, and the timer reset to a new offset.

    I checked the system event log for service starts in that window:

    9:00:57 AM  Windows Modules Installer service entered running state
    9:00:59 AM  Windows Update service entered running state  
    9:01:28 AM  Software Protection service entered running state
    9:01:28 AM  AppX Deployment Service entered running state

    Routine Windows Update maintenance. Something had restarted between those two timestamps, and the new instance picked up a new start-time anchor. Whatever was driving the hourly query was tied to a service that restarts, not to a fixed clock.

    That helped explain the cadence drift. It didn’t yet explain the cause. But it did confirm I was looking at a long-lived background process, not a scheduled task. Scheduled tasks fire on absolute clock times. Long-lived processes that do something hourly fire on offsets relative to their own start time, and reset when they restart.

    That was the right shape of thing to look for.

    Reading the tea leaves

    I went back and looked more carefully at the queries Pi3 was logging immediately before each aa.ns.charter.com lookup.

    19:19:33  query[SOA]  x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.x.0.0.a.4.0.0.1.0.0.4.c.6.0.0.6.2.ip6.arpa from 192.168.2.11
    19:19:33  forwarded to 127.0.0.1#5335
    19:19:33  reply is NODATA
    19:19:33  query[A]    aa.ns.charter.com from 192.168.2.11
    19:19:33  blocked upstream with NULL address aa.ns.charter.com is 0.0.0.0
    19:19:33  query[AAAA] aa.ns.charter.com from 192.168.2.11

    The Charter query wasn’t appearing in isolation. It was the second step in a sequence. The first step was an SOA query for a name in ip6.arpa.

    ip6.arpa is the reverse-DNS namespace for IPv6 addresses. When you want to ask “what hostname owns this IPv6 address?”, you construct a reverse-lookup name by reversing the nibbles of the address and appending .ip6.arpa, then ask for a PTR record.

    The name SKYE was asking about decoded to an IPv6 address in Charter Communications’ 2600:6c00::/24 allocation, the public space their residential service uses.

    So the query running through SKYE was a reverse-DNS lookup on a Charter IPv6 address. The query chain was traversing the ip6.arpa delegation, eventually landing at Charter’s authoritative nameservers for that block. And somewhere in that exchange, my DNS server ended up resolving aa.ns.charter.com. I did not yet know why.

    That reframed everything. I wasn’t looking for “a process that queries Charter.” I was looking for “a process that does reverse DNS on an IPv6 address in Charter’s space.”

    I checked my AD-integrated DNS zone for any records with IPv6 addresses in that range:

    powershell

    Get-DnsServerResourceRecord -ZoneName "howells.lan" -RRType AAAA
    HostName    RecordType  IP
    --------    ----------  --
    CHELLE-PC3  AAAA        2600:6c40:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx
    MIKE-PC3    AAAA        2600:6c40:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx

    There they were. Two Windows machines on my LAN had registered their public IPv6 addresses into my internal AD DNS zone, and those addresses were in Charter’s space. The client machines were periodically trying to register reverse PTR records for their Charter IPv6 addresses, and the domain controllers were forwarding the nameserver lookups that registration requires.

    Now the question was: what’s actually broken about Charter’s infrastructure that makes this generate a 0.0.0.0 response?

    The Charter trail

    I traced the delegation chain from the roots down to Charter, looking for where aa.ns.charter.com enters the picture.

    $ dig -x 2600:6c00:0:0::1 +trace +nodnssec
    ip6.arpa.       NS  f.ip6-servers.arpa.
                    NS  c.ip6-servers.arpa.
                    NS  a.ip6-servers.arpa.
                    ...
    0.6.2.ip6.arpa. NS  r.arin.net.
                    NS  u.arin.net.
                    NS  x.arin.net.
                    ...
    c.6.0.0.6.2.ip6.arpa.  NS  auth4.charter.com.
                           NS  auth3.charter.com.
                           NS  auth1.charter.com.
                           NS  auth2.charter.com.

    The delegation looked clean. IANA points to ARIN’s nameservers for the 6.2.ip6.arpa parent. ARIN delegates c.6.0.0.6.2.ip6.arpa (Charter’s 2600:6c00::/24 reverse zone) to auth1–auth4.charter.com. Those are Charter’s current authoritative nameservers, and they responded authoritatively when I queried them.

    So where was aa.ns.charter.com entering the chain?

    I asked Charter’s authoritative servers directly for the SOA record on that reverse zone:

    $ dig @auth1.charter.com SOA 0.4.c.6.0.0.6.2.ip6.arpa +noall +answer
    0.4.c.6.0.0.6.2.ip6.arpa. 14400 IN SOA aA.ns.charter.com. hostmaster.charter.com. 2019081361 7200 3600 604800 14400

    There it was.

    The SOA record, the Start of Authority, the record that identifies the primary master nameserver for a zone, names aA.ns.charter.com as the primary. That hostname has been decommissioned for years. It resolves to 0.0.0.0, presumably as a deliberate null route so that any client still trying to use it as a nameserver fails fast and stays failed.

    But the SOA itself still references it. And the serial number is 2019081361, which decodes as August 13, 2019. Charter hasn’t updated this SOA in nearly seven years.

    This was the cause, though not for the reason I first thought. My machines were not reading that SOA out of curiosity. They were trying to register a reverse PTR record for their Charter IPv6 addresses, and that is a dynamic DNS update. The protocol for those updates is RFC 2136, and it says the update has to go to the zone’s primary master. To find the primary master, the updater first asks for the zone’s SOA, then reads the MNAME (primary
    master nameserver) field, which names it. The MNAME on this zone is aa.ns.charter.com. So my DNS server resolves aa.ns.charter.com to get an address to send the update to. Charter returns 0.0.0.0. The update has nowhere to go, and Pi-hole sees the 0.0.0.0 answer and flags it as blocked.

    The bug is small: a stale hostname in one field of one SOA record at one ISP. But the chain of consequences from that one field to my Pi-hole UI showing dozens of “blocked” entries per day is a good example of how DNS quietly breaks in ways that nobody local to the problem can see.

    Why did Charter do this?

    The most likely answer is “Charter probably did update their delegation, but not their zone file.”

    When Charter restructured their authoritative DNS infrastructure at some point in the 2010s, they correctly updated the delegation at ARIN to point at their new nameservers (auth1–auth4.charter.com). And they did the careful thing with the old hostnames: instead of deleting them entirely, which would have caused legacy clients to retry NXDOMAIN aggressively, they pointed them at 0.0.0.0. Any client still trying to use them as a nameserver would fail closed in a stable, predictable way.

    That’s actually good engineering. The trap is that they forgot to update the SOA MNAME field inside the zone itself. The zone file still names aa.ns.charter.com as primary master. Most clients never read the MNAME. It is metadata that only matters when
    something tries to update the zone. But anything attempting a dynamic update, like a Windows machine registering a reverse PTR record, has to resolve the MNAME to find where to send the update. And when it does, it generates exactly the noise I was seeing.

    Charter probably has no idea. The bug is silent from their side. It produces no errors, no customer complaints, no operational impact. The only people who notice are people running their own recursive resolvers in environments that happen to have AD-integrated AAAA records in Charter’s address space, and who happen to look at their DNS logs carefully enough to wonder what aa.ns.charter.com is doing there.

    What I missed

    At this point I thought I had the whole story. I drafted a bug report to Charter’s NOC and sent it. I was going to wrap up the investigation.

    But I had one nagging question. The hourly queries were only ever from SKYE. Not from BOYD. Not from EMMA. Why only one of three otherwise-identical DCs?

    I dug through every possible difference I could think of. FSMO roles. OS versions. Scheduled tasks. DNS server cache contents. Scavenging settings. Stub zones. Conditional forwarders. Nothing distinguished SKYE from the other two DCs in any way that would explain it.

    Eventually I realized I’d been looking at the wrong data. I had been searching Pi3’s query log. But BOYD’s DNS forwarders point exclusively to Pi4, not Pi3. If BOYD was generating the same queries, they’d be on Pi4, not Pi3.

    I installed sqlite3 on both Pis and ran the same query against both pihole-FTL databases:

    sql

    SELECT datetime(timestamp,'unixepoch','localtime') AS time,
           client, domain, status 
    FROM queries 
    WHERE domain = 'aa.ns.charter.com'
    ORDER BY timestamp DESC 
    LIMIT 15;
    Terminal output from a sqlite3 query against Pi4's pihole-FTL database showing fifteen hourly aa.ns.charter.com queries from client 192.168.2.10.
    The same query, this time run against Pi4 instead of Pi3. BOYD (192.168.2.10) had been producing the same hourly Charter lookups the entire time. I’d just been looking at the wrong Pi-hole, and I hadn’t yet worked out which process on the domain controller was responsible.

    The combined picture was completely different from what I’d been seeing, and not in the direction I expected.

    BOYD was in the Pi4 log doing the same thing SKYE was doing on Pi3. Hourly. Different offset, :00:50 instead of :19:33, but the same pattern. So my first instinct was that I’d undercounted. It wasn’t only SKYE. It was SKYE and BOYD, two domain controllers each producing this query on its own cycle. I’d been fooled by looking at one Pi-hole at a time. That instinct turned out to be close to right, which surprised me, because my next move was to talk myself out of it.

    That instinct was also wrong, and it was the same mistake wearing a different coat.

    So I went the other way and decided the DCs were not really the source at all. Every blocked entry in both Pi-hole logs lists a domain controller in the client column, and I had read that column, all the way through this investigation, as “the domain controller is asking.” A workstation owns the record that starts this, its own Charter IPv6 address registered into my AD zone, so I leapt to the idea that the DC was just a passive relay for the workstation’s own lookup. That was the wrong correction. The workstation owns the record, but it is the DC’s own resolver that goes out and walks the chain. The Pi-hole only ever sees the DC because the DC is the one making the outbound query, not because the DC is forwarding someone else’s.

    I only sorted out which machine plays which role by capturing on both ends instead of staring at one. The workstation kicks the process off, it wants to register a reverse PTR record for its own Charter IPv6 address, and that registration has to find the zone’s primary master first. But the DC is the one that performs that lookup. Its resolver walks the SOA, reads the stale primary master, lands on aa.ns.charter.com, and gets 0.0.0.0, which is the exact sequence the Pi-hole logs show against the DC. The workstation starts it. The DC is the one knocking on Charter’s door.

    So it was two domain controllers, not three. BOYD and SKYE both do it. EMMA does not, and that is not an accident of where I was looking. When I enabled diagnostic logging on all three, EMMA’s log stayed empty and its cache held nothing for this name across the whole window, while BOYD and SKYE both lit up. The trigger is a single workstation, MIKE-PC3, whose public Charter IPv6 address lives as an AAAA record in my AD-integrated zone and replicates to every DC. CHELLE-PC3, the only other machine with such a record, has been powered off for weeks, so it registers nothing. One workstation owns the record. Two domain controllers act on it. EMMA, holding the same replicated record, sits the whole thing out, and I still cannot tell you why it and not the other two.

    I’d written the email to Charter saying “two Windows Server DNS recursors generate this query pattern hourly.” As it turns out that sentence was right, almost by accident. It is two, BOYD and SKYE, and they are recursors generating the query, exactly as written. I just did not actually know that when I wrote it, and I spent the next stretch of the investigation talking myself out of the correct answer before a packet capture talked me back into it. The bug I was reporting to Charter is real and unchanged either way.

    What I’m doing about it

    Nothing.

    The behavior is harmless. My DCs work. My DNS resolution works. The queries get returned promptly. The only visible artifact is a few dozen log entries per day that look like blocks but aren’t really blocks in any meaningful sense.

    I could remove the AAAA records for the affected machines from my AD zone, which would stop the trigger. I could disable IPv6 DNS registration on those clients, which would stop the records from coming back. But both of those are fixes for a problem that isn’t actually causing harm, and they’d touch configuration I’d rather leave alone.

    The right fix is upstream, at Charter. If they ever do it, the queries stop and I never have to think about this again.

    I sent them the bug report. I don’t have high hopes. The ARIN POC record for their NOC indicates they haven’t responded to ARIN’s own validation requests since 2020, which doesn’t suggest a particularly responsive inbox. But the email is in. We’ll see.

    For any Charter engineer who finds this

    If you work at Charter and you stumbled onto this post via a search for aa.ns.charter.com, here’s the fix: the SOA MNAME on the c.6.0.0.6.2.ip6.arpa reverse zone (and possibly other reverse zones in your IPv6 space) needs to be updated from aa.ns.charter.com to one of your current authoritative nameservers, probably auth1.charter.com. That’s a one-line change in a zone file. It will stop strict recursive resolvers around the internet from generating low-grade noise traffic against your 0.0.0.0 sinkhole every time they do reverse DNS on an address in your space.

    No urgency. Nothing is broken. But it would be a nice cleanup.

    Why I think this is worth writing up

    A few things stayed with me. The bug is invisible from Charter’s side. Their systems are fine, their customers are fine, nobody is going to file a ticket about this. The only people who notice are people running their own recursive resolvers in environments that happen to trigger the chain. That’s a specific and small population, and there’s no path from inside Charter to ever finding the bug. It only gets found from outside.

    The other thing is how many times I was wrong before I was right. I was wrong about the blocklist. I was wrong about SIL. I was wrong about “only SKYE,” then I overcorrected into “all three DCs,” then I overcorrected again into “the DCs are just relays for a workstation.” That last one is the one that stings, because I talked myself out of an answer that was basically correct. The DCs really were the ones generating the query, two of them, and I had to capture the recursion on the servers themselves before I would believe it. Each wrong turn took real effort to rule out, and the last one took a packet capture to settle. None of that effort was wasted, but it’s worth saying out loud. In this kind of work, most of your theories are wrong, and the job is proving them wrong methodically until the right one is left.

    I started this expecting a five-minute mystery. It turned into a few hours of investigation, a bug report to a major ISP, and a blog post.

    The line I noticed at the start is exactly the kind of thing most people would scroll past. I almost did. I’m glad I didn’t.

    If you’ve got a Pi-hole and an unfamiliar entry in the log: pull the thread. You might be surprised what’s on the other end.

    One thing I could not nail down

    There is a single piece of this I never resolved, and I want to flag it honestly rather than pretend the case is airtight.

    The failed lookup repeats on a roughly hourly cycle. I confirmed that much directly, on both BOYD and SKYE, in the DNS Server’s own recursion log. What I could not find is any documented Windows interval that explains the hour, or even pin down which timer inside the DNS Server is driving it.

    I checked the obvious candidates and ruled them out by measurement. It is not a DHCPv6 lease, because the network is SLAAC only. It is not the 24-hour DefaultRegistrationRefreshInterval, which is absent on the machine. It is not exponential backoff, because the interval is flat, not lengthening. I also ran the question past several other analysis tools. The consensus was that no public Microsoft document names a one-hour retry interval for a failed dynamic registration, and the one source that claimed otherwise turned out to be fabricated and was retracted when I asked for the link.

    My current best guess is that the hour is not a retry interval specific to this failing record at all. What kicks it off is a workstation doing a full registration pass, the IPv6 reverse and the IPv4 reverse together, not a targeted retry of the one broken record. So I suspect what I am watching is a periodic re-registration sweep, and the doomed IPv6 PTR simply rides along on every pass. That would also explain why it never backs off. A dumb sweep does not reason about individual records. It just re-runs the whole set. What I cannot account for is why two of my three domain controllers act on that sweep and the third, holding the same replicated record, never does.

    But that is inference, not proof. If you actually know, I would genuinely like to hear it. Is there a documented periodic interval inside the Windows DNS Server that would make its recursion engine re-walk this reverse name about once an hour, and is there a known reason one domain controller would sit out a sweep its peers act on. If you have a source, leave a comment. I will update the post and credit you, the same way I did with the RFC correction earlier.

    Update, June 25. I went back and captured this directly instead of leaving it at inference. With DNS Server diagnostic logging enabled on all three domain controllers, BOYD’s recursion log caught the full chain on our exact name. First the reverse-PTR query for the workstation’s Charter IPv6 address, then an SOA lookup that returns the stale primary master, then the resolution of aa.ns.charter.com, which comes back 0.0.0.0. SKYE shows the same chain. EMMA’s log stayed empty. So the mechanism in this post is captured rather than assumed. The domain controller’s own DNS Server is the actor, the workstation only owns the record that triggers it, and it is two of the three DCs, not one machine and not all three. The one piece still open is the hourly timer, which is the section above.

    This bug report is currently open with Charter Communications’ NOC. If they respond or fix it, I’ll update this post. If they don’t, the post stands as a documented walkthrough for the next person who sees aa.ns.charter.com in their query log and wonders what it is.

    Correction. An earlier version of this post said that Windows resolves the SOA’s MNAME for cache validation, and attributed that to RFC 1034. Patrick Mevzek pointed out that 1034 says no such thing, and he is right. Here is why I reached for it and why it was wrong. The behavior involves an SOA, and RFC 1034 is where the SOA and DNS caching are defined, including an optional, not-recommended scheme for revalidating cached data. But that scheme keys on the SOA’s SERIAL field, not the MNAME, and it never resolves anything. The MNAME lookup is not a caching step at all. It is the first move of a dynamic update under RFC 2136. My two machines were trying to register reverse PTR records for their Charter IPv6 addresses, and RFC 2136 sends an update to the zone’s primary master, which the updater finds by resolving the SOA’s MNAME. That MNAME is aa.ns.charter.com, Charter null-routes it to 0.0.0.0, and Pi-hole logs the 0.0.0.0. Right record, wrong purpose, wrong RFC. Thanks to Patrick for the catch.

    Fediverse Reactions
  • Deploying DNS over HTTPS on Windows Server 2025, and the Three Things the Docs Skip

    Deploying DNS over HTTPS on Windows Server 2025, and the Three Things the Docs Skip

    DNS has carried our most basic network lookups in cleartext for its entire life. On June 9, 2026, Microsoft moved DNS over HTTPS for the Windows DNS Server role to general availability, shipping it in the June cumulative update (KB5094125) for Windows Server 2025. That means the encrypted, authenticated client-to-resolver path that used to require a separate appliance or a public resolver now lives inside the DNS role you already run.

    I rolled it out across three domain controllers a few nights after it went GA and ran a full verification pass rather than just trusting the switch. This post is that deployment, start to finish, with the evidence that proves it is genuinely encrypting traffic. It also covers the three things the official walkthrough does not warn you about: a certificate enrollment trap, a verification command that will quietly lie to you about whether DoH is working, and a Settings page that insists DoH is off when it is actually on.

    What this does, and what it does not

    DoH encapsulates DNS queries and responses inside HTTPS, encrypted with TLS, and uses the server certificate to authenticate the resolver to the client. The practical wins are the obvious ones: queries are no longer readable by anyone passively watching the wire, and a client can verify it is talking to the resolver it expects rather than an impostor.

    Scope, stated up front The single most important thing to understand before you deploy is the scope. This GA release encrypts the client-to-server leg only. The path from your DNS server out to its upstream forwarders, and the traffic between domain controllers, both remain on traditional plaintext UDP 53. Microsoft has stated that encrypted communication to upstream resolvers is a planned future update.

    So calibrate your expectations accordingly. You are encrypting the hop between your clients and your DNS servers. You are not, with this feature, encrypting the path from your DNS servers out to the internet. If that external hop matters to you, it likely needs a separate mechanism (an encrypted forwarder, for example). In my own environment that outbound hop was already encrypted by the forwarders, so this release closed the one internal leg that was still in the clear.

    Prerequisites

    • Windows Server 2025 with the June 2026 cumulative update (KB5094125) or later. The DoH role surface does not exist on earlier builds.
    • A certificate on each DNS server that meets four requirements: a Server Authentication EKU (1.3.6.1.5.5.7.3.1), a Subject Alternative Name matching the hostname or IP you will put in the DoH URI template, a private key present in the Local Computer store, and issuance from a CA that both the DNS server and the clients trust. An internal enterprise CA makes this trivial, but a public certificate works just as well.
    • A firewall rule allowing inbound TCP 443 on each DNS server.
    • Administrative access to each DNS server.

    The deployment, step by step

    I worked one server at a time and fully verified each before moving to the next. If your DNS servers are domain controllers, do the one holding the most fragile dependencies last, since the DNS service restart at the end briefly interrupts resolution on that box.

    1. Get a certificate onto each DNS server

    There is a subtlety here that bites people, covered in the first gotcha below. The short version: if you want the private key created directly in the machine store and never written to disk, request it on the server in the machine context. With an enterprise CA and a published server-auth template, that is a one-liner:

    PowerShell · request the cert in the machine store
    Get-Certificate -Template WebServer `
  -SubjectName "CN=dc01.corp.example.com" `
  -DnsName "dc01.corp.example.com" `
  -CertStoreLocation Cert:\LocalMachine\My

    Confirm the result has the private key and the right SAN:

    PowerShell · confirm private key and SAN
    Get-ChildItem Cert:\LocalMachine\My |
  Where-Object Subject -match "corp.example.com" |
  Select-Object Subject, Thumbprint, NotAfter, HasPrivateKey,
    @{n='SAN';e={($_.Extensions | Where-Object {$_.Oid.FriendlyName -eq 'Subject Alternative Name'}).Format(0)}}

    You want HasPrivateKey set to True and the SAN showing the FQDN you will use in the URI template.

    2. Bind the certificate to the HTTPS listener

    PowerShell · bind the cert to the HTTPS listener
    $guid = New-Guid
netsh http add sslcert ipport=0.0.0.0:443 `
  certhash=<your-cert-thumbprint> appid="{$guid}"

    If you would rather DoH answer on one specific address instead of all of them, replace 0.0.0.0 with that IP. It has to match, or resolve to, the host in your certificate SAN. Confirm the binding landed:

    PowerShell · confirm the binding
    netsh http show sslcert

    A quick note in case you see it: if netsh returns Error 183, “Cannot create a file when that file already exists,” it simply means a binding on that address and port is already present. The operation is idempotent, so a re-run reports the existing binding rather than breaking anything.

    3. Allow inbound TCP 443

    PowerShell · allow inbound TCP 443
    New-NetFirewallRule -DisplayName "DNS over HTTPS" -Direction Inbound `
  -Protocol TCP -LocalPort 443 -Action Allow

    If you bound DoH to a non-default port, substitute it here, and remember any upstream hardware firewall needs the same allowance.

    4. Enable DoH and set the URI template

    PowerShell · enable DoH and restart the service
    Set-DnsServerEncryptionProtocol -EnableDoh $true `
  -UriTemplate "https://dc01.corp.example.com:443/dns-query"

Restart-Service -Name DNS

    The port in the URI template has to match the port you bound the certificate to.

    5. Verify the listener actually started

    PowerShell · verify the listener
    Start-Sleep -Seconds 5
Get-DnsServerEncryptionProtocol

    You want EnableDoh set to True with your URI template echoed back. The Start-Sleep is not decoration. Query that cmdlet within about a second of the DNS restart and it throws a WIN32 21 (ERROR_NOT_READY), because the management provider has not finished coming back up. Give it a few seconds and it answers cleanly.

    The authoritative confirmation is in the event log. Open Event Viewer, go to Applications and Services Logs, then DNS Server, and look for Event 822:

    Event Viewer · DNS Server log · Event 822
    Id          : 822
Message     : Successfully started HTTP server for DNS-over-HTTPS (DoH) server.
              The DoH server is listening on following URL(s):
              https://dc01.corp.example.com:443/dns-query

    Event 822 is the line that separates “configuration accepted” from “HTTPS listener actually running.” If you see events in the 823 to 826 range instead, those are initialization failures, and the error code in the message is where to start.

    Across all three servers the listener came up clean:

    DNS server URI template Listener (Event 822)
    DC01
    10.0.0.10 · primary, PDC emulator    		 https://dc01.corp.example.com:443/dns-query CONFIRMED
    DC02
    10.0.0.11 · secondary    		 https://dc02.corp.example.com:443/dns-query CONFIRMED
    DC03
    10.0.0.12 · Server Core    		 https://dc03.corp.example.com:443/dns-query CONFIRMED
    Gotcha 1: enrolling as the machine account gets denied by the default template Requesting the certificate into Cert:\LocalMachine\My runs the enrollment in the context of the machine account (for example, DC01$), not your user account. The stock Web Server template grants Enroll only to Domain Admins and Enterprise Admins, so the machine account is not on the list, and every request comes back with:
    Get-Certificate · first attempt
    CertEnroll::CX509Enrollment::Enroll: You do not have permission to request
this type of certificate. 0x80094012 (CERTSRV_E_TEMPLATE_DENIED)
    This is easy to misread as a broken request. It is not. It is the template ACL doing exactly what it says. The clean fix is to grant the machine the right to enroll: open the Certificate Templates console, edit the Web Server template, and on the Security tab add the Domain Controllers group (or whatever group covers your DNS servers) with Read and Enroll. Two reasons this is the right move rather than a workaround. First, it is the correct, durable answer for machine-context server-auth enrollment. Second, it turns future certificate renewals into one-line commands instead of a repeat of this dance. If your DNS servers are domain controllers, granting them enroll rights on a server-auth template is a non-issue from a blast-radius standpoint, since they already are the trust anchor of the domain. If a server still reports denied immediately after the change, its local enrollment-policy cache has not refreshed yet. Run certutil -pulse on that server and retry.

    Configuring a client

    The server side answering DoH does nothing on its own. DoH is opt-in on the client, and the client will keep using plaintext 53 until you tell it otherwise. There is also a gate: a Windows client will only use DoH for a DNS server that is on its list of known DoH servers. The public resolvers ship on that list by default, but your own servers do not, so you register them first.

    PowerShell · register the DCs as known DoH servers
    Add-DnsClientDohServerAddress -ServerAddress '10.0.0.10' `
  -DohTemplate 'https://dc01.corp.example.com/dns-query' `
  -AllowFallbackToUdp $True -AutoUpgrade $True
# repeat for each DNS server (DC02, DC03, ...)

    That is all the client needs. Those entries, with AutoUpgrade set to True, are what actually upgrade your queries to DoH, and ipconfig /all will now annotate each server with its template and fallback state as confirmation. You can also configure this through the Settings GUI instead of PowerShell, but the way the GUI and PowerShell relate to each other has a genuinely confusing catch that deserves its own treatment. If you configure with PowerShell as shown here, do not be alarmed when the Settings page still shows the servers as unencrypted. That is expected, and a dedicated section below explains exactly why, and what each GUI option does if you decide to change it.

    ipconfig /all, client adapter (excerpt)
    DNS Servers . . . . . . . . . . . : 10.0.0.10
                                      DoH: https://dc01.corp.example.com/dns-query
                                      unencrypted fallback
                                    10.0.0.11
                                      DoH: https://dc02.corp.example.com/dns-query
                                      unencrypted fallback
                                    10.0.0.12
                                      DoH: https://dc03.corp.example.com/dns-query
                                      unencrypted fallback
    Do not set Require DoH on a domain-joined machine Windows offers an Encrypted only, or Require DoH, setting, and Microsoft own client documentation explicitly warns against using it on domain members. Active Directory depends entirely on DNS, and the DC-to-DC and replication paths are not DoH. If you require encrypted DNS and the encrypted path has any hiccup, you do not harden the domain, you break name resolution for it. Encrypted preferred with UDP fallback keeps resolution resilient while still using the encrypted path whenever it is available. If you genuinely need the internal AD DNS traffic encrypted end to end, the supported route is IPsec connection security rules, not Require DoH.

    Proving it is actually encrypted (and the gotcha that hides it)

    Here is the trap that cost me a few minutes, and it is worth your attention because the obvious test gives a false result. With UDP fallback allowed, a successful lookup proves nothing on its own, because the same answer comes back whether the query went over DoH or plaintext 53. The client configuration proves intent. Only the server-side counter proves the traffic.

    My first measurement read flat zero, and the cause was the test, not the deployment. I was forcing queries at a specific server with Resolve-DnsName -Server <ip>. The problem is that specifying -Server explicitly overrides the interface DoH path and issues a direct query, which leaves on plaintext 53 and never touches the DoH listener. The counter sat at zero across every sample:

    DoH Requests Received/sec · during the -Server run
    doh requests received/sec : 0
doh requests received/sec : 0
doh requests received/sec : 0
...twelve consecutive zeros...

    Dropping -Server entirely and generating real system-resolver traffic instead (a cache-busted loop of ordinary web requests, which routes through the configured DoH path) lit the counter up immediately:

    DoH Requests Received/sec · during real resolver traffic
    doh requests received/sec : 0.4999
doh requests received/sec : 0.4987
doh requests received/sec : 0.9973
doh requests received/sec : 1.4963
...nonzero throughout...

    You can watch this live on the DNS server while a client generates lookups:

    PowerShell · watch the counter live
    Get-Counter -Counter "\DNS-over-HTTPS\DoH Requests Received/sec" -SampleInterval 2 -MaxSamples 15

    That counter measures encrypted DoH packets separately from traditional DNS, so any nonzero reading is unambiguous proof the client is on the encrypted path. If you want a record that does not depend on catching the counter live, enable the DNS Server Analytical log and look for events 597 (encrypted query received) and 598 (encrypted response sent). Both carry a Channel value of 2 for DoH, and the 598 event includes the HTTP status, so a single 598 showing HTTP/2 with Status 200 is durable, timestamped proof.

    What counts as proof A successful lookup does not prove encryption while fallback is allowed. A nonzero DoH counter does, because it tallies encrypted packets separately from plaintext DNS. For a record you do not have to catch live, a single Analytical event 598 with Channel 2 and an HTTP/2 200 status is timestamped proof.

    The lesson worth keeping: never validate DoH with an explicit-server query. It will tell you encryption is off when it is actually on.

    Why the Settings app shows “Off” when DoH is actually on

    This is the part that nearly convinced me my own deployment had failed, and it is the most confusing thing about configuring DoH this way, so it gets its own section.

    If you configure the client with PowerShell as shown earlier and then open Settings to check your work, here is what you will see: every DNS server listed as “Unencrypted,” and the per-adapter “DNS over HTTPS” dropdown set to “Off.”

    Windows 11 Edit DNS settings dialog showing both DNS over HTTPS dropdowns set to Off while DoH is actually active through PowerShell

    Your instinct will be that it did not work. It did. DoH is running. The Settings app is simply not showing it, and understanding why means knowing that Windows keeps this configuration in two separate places:

    • The system-wide known-server table. This is what Add-DnsClientDohServerAddress writes to. When an entry has AutoUpgrade set to True, that entry is what actually upgrades your queries to DoH. This layer does not appear anywhere in the basic Settings view.
    • The per-adapter dropdown in Settings. This is a separate, GUI-managed setting attached to the network interface. If you configured DoH through PowerShell and never touched this dropdown, it stays on “Off.”

    The label you see in Settings reads only the second layer. So it will report “Off” and “Unencrypted” even while the first layer is encrypting every query you send. In the display, the two layers do not know about each other.

    To see the layer that is actually in effect, run:

    PowerShell · show the known-server table
    Get-DnsClientDohServerAddress
    Get-DnsClientDohServerAddress · output
    ServerAddress   AllowFallbackToUdp AutoUpgrade DohTemplate
-------------   ------------------ ----------- -----------
10.0.0.12       True               True        https://dc03.corp.example.com/dns-query
10.0.0.11       True               True        https://dc02.corp.example.com/dns-query
10.0.0.10       True               True        https://dc01.corp.example.com/dns-query
1.1.1.1         False              False       https://cloudflare-dns.com/dns-query
1.0.0.1         False              False       https://cloudflare-dns.com/dns-query

    Read it this way: for each of your servers, AutoUpgrade set to True means DoH is active for that server, and AllowFallbackToUdp set to True means it will fall back to plaintext if encryption is unavailable. Those entries are your real configuration. The built-in public resolvers below them sit at False because you never opted them in, which is also why they, and everything else, show as “Unencrypted” in the GUI: that label is tracking the dropdown, not this table.

    Here is the proof that these really are two separate stores. If you flip the GUI dropdown from Off to On (automatic template), then run Get-DnsClientDohServerAddress again, the output comes back byte for byte identical. The GUI change does not appear in it at all. That is because this cmdlet reports the global known-server table, while the dropdown writes a separate per-adapter setting that this view never surfaces. So no single view tells you the whole story: this cmdlet shows the global auto-upgrade policy, the Settings page shows a per-adapter preference, and neither one reflects the other.

    The only thing that settles it beyond any label is the server-side counter from the previous section. If that counter moves while a client resolves names, DoH is flowing, regardless of what Settings claims.

    What each dropdown option means, and what changing it does

    Because the dropdown is sitting right there reading “Off,” you will be tempted to change it. Here is exactly what each choice does and how it interacts with the PowerShell table you already set.

    The dropdown has three states:

    • Off. The per-adapter DoH setting is off. The important part: this does not disable your PowerShell AutoUpgrade entries. With the table configured as above, DoH keeps working even with the dropdown on Off. That gap is the entire source of the confusion.
    • On (automatic template). Turns on DoH for the adapter and pulls the template automatically from the known-server list for that server. The template is simply the DoH URL, the https://server/dns-query address that the server answers encrypted queries on. Use this when the server is already in the known list, which yours are, so Windows can look the URL up for you.
    • On (manual template). The same, except you type that DoH URL yourself rather than letting Windows find it. Use this for a resolver that is not in the known list, or when you want to pin a specific URL.

    When you pick either “On” option, a separate Fallback to plaintext toggle appears, and that toggle is the setting that actually matters:

    • Fallback to plaintext on gives you “encrypted preferred.” The client tries DoH and falls back to plaintext if it cannot. This is the safe choice.
    • Fallback to plaintext off gives you “encrypted only,” also known as Require DoH. The client refuses to resolve at all if DoH is unavailable.

    So if you want the Settings page to stop saying “Off” and instead reflect what is happening, set the dropdown to “On (automatic template)” and leave “Fallback to plaintext” turned on. The summary will then read “Encrypted,” and you will have expressed through the visible layer the same intent the table was already enforcing. Changing the dropdown does not delete your PowerShell entries. The two layers simply agree now instead of appearing to disagree.

    Do not turn Fallback to plaintext off on a domain-joined machine Choosing “encrypted only,” or Require DoH, on a domain member is the one genuinely dangerous setting here. Active Directory depends entirely on DNS, and the domain controller to domain controller paths are not DoH. If you require encryption and it becomes unavailable for any reason, the client stops resolving names, and on a domain member that means it can no longer find the domain. Microsoft recommends against Require DoH for domain-joined computers for precisely this reason. Keep fallback on.

    The short version

    If you remember one thing from this section: the Settings app does not reflect DoH that was configured with PowerShell, and PowerShell does not reflect what you change in Settings. They write to different places and neither shows the other. So pick one method and stay with it. PowerShell is the repeatable, scriptable choice, it is what works cleanly across more than one machine, and it is what I would treat as the source of truth. Whichever you choose, a reading of “Off” or “Unencrypted” in Settings means nothing on its own, Get-DnsClientDohServerAddress shows you the known-server table but not the per-adapter setting, and the server-side counter is the only thing that actually proves encryption is happening. Configure once, deliberately, and verify with the counter rather than chasing the GUI label.

    Where the encryption actually sits

    It helps to picture the whole resolution chain and mark which legs this change touches.

    DNS resolution chain showing the client to DNS server leg now encrypted with DoH, the DNS server to forwarder leg still plaintext on the LAN, and the forwarder to internet leg already encrypted
    Two of three legs encrypted. The remaining plaintext hop stays inside the switch and is the one Microsoft has slated for a future upstream-encryption update.

    The client-to-server leg is now encrypted and authenticated. The server-to-forwarder leg stays plaintext until Microsoft ships upstream encryption, but in a typical setup that hop never leaves your own switch, and the forwarder-to-internet hop can already be encrypted independently. So this closes the internal leg that was exposed, without touching the parts that were already handled.

    Is it worth doing?

    Be honest with yourself about the threat model. This is defense in depth on an internal segment. The realistic attacker it stops is one who is already on your LAN, passively reading or tampering with DNS between your clients and your servers. If that is not in your threat model, the security gain is modest.

    That said, the cost is genuinely low. If you already run an enterprise CA, the certificate is a one-liner, the enablement is a handful of commands, and the client side is a registration plus a dropdown. It aligns your name resolution with Zero Trust principles, it is hands-on practice with a feature that is brand new to the platform, and it closes a leg that was previously in the clear. For most people running Windows Server 2025 DNS, the answer is yes, with the single firm caveat that you never set Require on a domain-joined client.

    Wrapping up

    DNS over HTTPS on Windows DNS Server is straightforward to deploy once you sidestep the three traps: the machine-context enrollment that the default template denies, the explicit-server query that hides whether DoH is working, and the Settings page that reports DoH as off when it is on. Get the certificate right, enable it, configure clients to prefer encryption with fallback, and verify with the counter or the analytical events rather than a lookup or a GUI label that could be lying to you. The result is an encrypted, authenticated client-to-resolver path running inside the DNS role you already operate, with no new appliance and no architectural change.