One of the most common questions I hear from small and even medium-sized ISPs is ‘Why should I run my own DNS resolver(s)?’
The perception that the Domain Name System (DNS) is hard, complicated or even unnecessary is often cited as a reason to just farm it out to one of the ‘free’ anycast resolver services.
Now, there are many reasons to be wary of the DNS, both from the professional and the consumer side — it is a huge treasure trove of personal information about behaviour, and is easily monetized by entities large enough to consume and process it. I’ve even written about sidestepping your providers’ DNS all together (although my experiment, which lasted a bit over a year, was a tad more controlled than simply hacking DNS over HTTPS (DoH) into my system and moving to an anycast resolver service).
As an Internet Service Provider (ISP), there are so many reasons to run your own DNS (both recursive resolvers and authoritative systems). Just to rattle off a few, it enables you to:
- Control your PTR (reverse lookups) locally — you need these whether you know it or not.
- Log queries — very helpful for determining bad actor behaviour on your network.
- Use DNS RPZ — Response Policy Zones allow you to do a handful of things, including filtering known bad actor domains.
- Keep your data as local as you can.
Now, these definitely aren’t the only reasons, but they’re pretty important. I will focus on the last one here because it pertains to my crude little project I call the DNS latency monitor. Authoritative and recursive is a topic for another post in and of itself.
To understand where I am going, it is probably best to understand my perspective.
Broken DNS makes the network functionally broken for an average user
There is an old networking adage – “It’s always MTU, unless it’s DNS”.
The DNS is the single most non-networking protocol that causes the most networking pain. It will always be up to those of us looking after the network to prove that a DNS issue is not a networking problem, and this is true tenfold when it comes to ISPs.
The customer (or Carol in payroll) does not care what is broken, they just know something does not work. They can’t get to Google. They cannot get email. Netflix won’t stream. The Internet is broken.
DNS is like the Border Gateway Protocol (BGP); you have to have it for the Internet to work. Why anyone would want to outsource something with that high a visibility is beyond me, but I do understand that resources are hard to come by in small ISPs and that many blogs have encouraged the use of DoH for the end-user to sidestep provider based resolvers (and for those that don’t know, even Comcast provides a DoH option). In my opinion (and experience) this is a mixed bag of cats and razor blades.
Small providers aren’t likely to sell off your browsing data. The big providers that are using it are the ones you cannot change — cellular devices. While on cellular most if not all devices/providers do not allow you to change from their on-net resolvers. This is of particular interest for two reasons:
- At least one of the large US providers owns a company that literally analyses this data for use.
- Cellular devices comprise a huge amount of the active devices on the Internet (we’ll save the smug IPv6 network deployment talk about this fact for another day).
So really, an average user changing this data on their home network but still carrying around a smartphone is really closing a door but leaving the windows wide open. Yes, I do realize some apps account for this, but none are seamless and your mother is almost certainly not going to use them.
The ironic thing here is that the DNS is actually quite easy to run
It is true that in the early days, BIND was a bit of a bear (most of that has changed, and it’s still the most feature-rich).
That has largely changed with the introduction of so many solid DNS resolvers. Unbound, for example, is lightweight, very feature-complete, super well documented, and incredibly easy to install. It will also happily run on very low-powered machines.
Where is this meandering nonsense all going, you might ask? It’s going to the anycast DNS latency logging ‘service’ that came to life at the beginning of the US COVID-19 shelter-in-place order. I wanted to see what changed, if anything, from a customer quality of experience perspective.
What I learnt was that a single DNS query to an outside system can add a noticeable amount of latency RTT to every single request that a customer makes.
As a service provider not running your own recursive resolvers, imagine a 120ms pad on every web page requested. Now imagine that across every subscriber. Then imagine the anycast system closest to you goes down, and you’re hitting the next one over. Every single DNS request and response is now at the mercy of the Internet at large. Totally out of your control, because you neither control the entire path and have zero input as to how that service is being run.
So what’s the benefit?
Running your own resolver is straightforward, fairly easy, and lets you take back some of that control.
As a customer not wanting to use an ISP’s resolver, you can also consider that you’re freely handing over all of the DNS queries you make to a company that can and does (in some cases) monetize it.
So what does this thing really do? It’s quite simple, actually. It’s a run of the mill smokeping master-slave install. Why smokeping? Check out the project front page and it’ll explain.
To most closely mimic a user experience, it currently runs 16 DNS query tests for the same DNS name and reports the RTT back from the anycast resolver closest to it. Then smokeping does what smokeping does — it records it.
So far it has been a good litmus test for how the largest of the anycast recursive resolvers behave from any given point of view. Sometimes it’s great, sometimes it’s not.
This is useful data not just for the smaller ISPs that may be using them, but also for anyone trying to pick out and troubleshoot different performance or DNS issues. There are certainly commercial tools that do this, but none that I have seen are freely available to both use and participate in, while also providing a consistent, long-term interface into the data. I am, as always, happy to be corrected in that regard.
I plan to keep this running as long as folks keep providing test points. It’s a very lightweight install and is totally controlled by the master system, so once you get it installed, it’ll just run. Many of these probes run on Linux containers, raspberry pi systems, or other lightweight Unix environments. If you’re interested in providing a probe on the DNS latency monitor project, please reach out and contact me. I am currently looking for probes in:
- Hawai’i
- Guam
- Australia (both coasts and Tasmania)
- Singapore
- India
- Canada (anywhere)
- Iceland
- Greenland
- Mexico
- Any and all economies in Central and South America
- Japan
- Eastern Europe
- Ireland
- UK
- Any Caribbean islands
- Anywhere in Africa
If you’re ready to dive into running a set of DNS recursive resolvers for your environment, here are some simple instructions, where you can check out everything you should need to get a basic install up and running.
Adapted from original post which appeared on The ForwardingPlane
Nick Buraglio has worked in the network provider industry for more than 15 years, holding Network Engineering positions at Regional Internet Providers as well as at the National Center for Supercomputing Applications.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.