Recently I have been exploring improved home DNS services, with a view to blocking spam and adverts. I’ve been using the PI-hole and AdGuard systems to run my own in-house DNS, and redirect some known domain names to unrooted IP addresses. This has made things easier for myself and my home users, and so far seems to be working well. At least, it’s working well NOW. The journey to get here has not been entirely smooth.
In the home, things work until they don’t
Building out this system reminded me that, when I do this, I’m effectively standing in for a competent, paid systems administrator. The people in my house and the devices on my home network that I depend on all rely on me not to randomly turn things off or break them. The Digital Video Recorder, sound system, home file server that backs up the PCs every night, even a tiny GPS sensor I run for a European project monitoring GPS/GLONASS satellite visibility, all depend on me as the systems administrator.
Unfortunately, DNS systems sometimes need to be rebooted, and when that happens, everything else on the home network stops. Changing the router configuration can bring down the entire wireless network while it reloads. Changing the router’s DNS configuration affects DHCP, which in turn causes a freeze-and-thaw cycle across the network.
Changing things causes outages.
The failure wasn’t really the DNS. The DNS was simply the latest reminder that critical services become invisible once they’re working. We stop seeing them as infrastructure and start treating them as appliances. That’s usually when we discover we never built any resilience into them at all.
The answer is more resiliency through more redundancy
In this context, it’s easy to forget that, in the real world, at work, on my phone’s mobile network, and even at conference venues, these services aren’t run as single instances when they don’t have to be. There should be at least two, configured independently, and you should avoid rebooting both at the same time whenever possible.
Of course I have a backup plan: I just hadn’t tested it.
I’d love to say I had a backup plan and put it into action. The reality is that I had one in theory, but hadn’t actually implemented it. In moving my DNS to a new platform, I forgot to confirm there was an alternative in place before rebooting the primary. As things around the home network started to break, I discovered it wasn’t a primary at all. It was the only instance.
I’m now trying to reacquire the habits that apply in the workplace, making changes to my home network:
- Do I understand all the device dependencies related to the thing I am working on?
- Is there an alternative service that my home network will adopt if I change this one?
- If it turns out I’ve made a mistake, can I put things back the way they were before I decided to change them?
The interesting thing is that none of these questions is specific to the DNS. They’re the same questions I’d ask before making a change to any production system. The only difference is that I forgot my home network had quietly become one.
I don’t know that every home network needs enterprise-grade redundancy. Most of us can tolerate the occasional outage. What this experience reminded me of though is that resilience isn’t something you buy when you purchase hardware. It’s something you design into a system and then repeatedly test.
As homes accumulate more devices, services and dependencies, the distinction between a home network and a small production environment keeps shrinking. Unfortunately, the systems administrator is still me.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.