ZDNS: A fast DNS toolkit for Internet measurement

Active DNS measurement is fundamental to understanding and improving the DNS ecosystem. However, the absence of an extensible, high-performance and easy-to-use DNS toolkit has limited both the reproducibility and coverage of DNS research. 

To promote the scalability and reproducibility of DNS research, my colleagues and I at Stanford University have built ZDNS. ZDNS is a modular and open-source active DNS measurement framework optimized for large-scale research studies of DNS on the public Internet.

Key Points:
• ZDNS is an open-source scanning system that can resolve 50M domains in 10 minutes and query the PTR records of the full public IPv4 address space in 12 hours.
• ZDNS implements its own internal caching and recursion, thereby exposing internal DNS operations.
• ZDNS provides a simple command line interface for scanning and outputs results in programmatically interpretable JSON.

ZDNS guiding principles and architecture

ZDNS’ architecture fulfils four requirements: 

1. Exposed DNS lookup chain: Recursive resolvers hide many characteristics of DNS operations. Thus, ZDNS performs its own recursive resolution and exposes lookup chains in addition to supporting external recursive resolvers. ZDNS fully exposes DNS lookup chains by implementing its own DNS library. The ZDNS library provides recursive lookups, caching, validation, exposed transcripts of exchanged packets and easy DNS question construction and answers parsing.
2. Safe: Internet servers regularly respond with malformed responses due to misconfiguration and intentionally malicious operators. Thus, ZDNS is written in a memory-safe language, Go,  and supports a modular interface where researchers can easily and safely implement new functionality.
3. Easy to Use:  In order to promote adoption, ZDNS implements a framework for easy command-line interaction. The core framework is responsible for facilitating command-line configuration, spawning lookup routines, delegating control to modules, encoding, decoding and routing data to/from routines, load balancing against upstream resolvers, aggregating run-time statistics, and ensuring consistent output. 
4. Extensible: To allow users to easily extend ZDNS to meet their own needs, ZDNS implements composable modules.  Modules are responsible for providing DNS query-specific logic, including logic specific to a DNS query or record, expected query response, additional command line flags, and global and per-routine initialization. Developers add modules by implementing a simple interface. 

ZDNS performance optimizations 

DNS experiments frequently require querying a large number of names. 

Several optimizations are critical to ZDNS’s performance.

1. Parallelism: The majority of clock time spent making a DNS query is spent waiting for a response rather than constructing or parsing DNS packets. Thus, efficiently parallelizing a sufficiently large number of concurrent queries is crucial. We build ZDNS in Go so that we can use the language’s ability to efficiently manage thousands of concurrent queries using lightweight routines. 
2. UDP Socket Reuse: Creating a socket for every lookup is expensive because each socket is used to send and receive only two packets before being torn down and recreated.  Instead, we establish and maintain a single long-lived raw UDP socket in each lightweight routine for the lifetime of the program execution.
3. Selective Caching:  While popular recursive resolvers, like Unbound, cache results in order to quickly respond to lookups for frequently queried names, we expect ZDNS users to frequently lookup unique names. Thus, we selectively cache only NS and glue records to help with future recursion, but do not cache any results for the leaf names being directly queried.

Evaluating ZDNS

We evaluate ZDNS scalability, execution time, and success rate when performing billions of queries and compare ZDNS to a set of existing tools. For exact experiment configurations, please see the full paper. 

ZDNS performance scales when performing large amounts of consecutive lookups (Table 1). For example, ZDNS resolves the entire IPv4 address space in 12.1 hours when using Google’s public resolver and in 116.7 hours when using its own iterative resolver, while maintaining a success rate of 93% and 88.5%, respectively.

ZDNS success rate drops less than 2% when scaling from millions to billions of lookups.

Table 1 — ZDNS resolves 100% of the IPv4 address space in 12.1 hours using a public recursive resolver and in 116.7 hours using its own iterative resolver.

We compare ZDNS performance against three popular tools — Dig, Unbound, and MassDNS — when resolving 10M random IPs/domains. 

ZDNS is 85 times faster than Dig when exposing the DNS lookup chain. Since Dig was never designed to be a high-performance scanning engine, it achieves a peak performance of only 120 successful RR type A lookups per second when using Cloudflare’s resolver to perform a trace query. In contrast, ZDNS performs 10K RR type A lookups per second when using its own iterative resolution.

Compared to other existing higher-performance tools, ZDNS achieves 2.6 to 3.6 times more successful queries per second and up to 35% less packet drop (Table 2). 

For example, while ZDNS and Unbound achieve the same number of successes, the ZDNS iterative resolver successfully resolves 2.6 to 3.6 times more names per second.

Table 2 — ZDNS iterative resolver performs 2.6 to 3.6 more successful queries per second than Unbound, and experiences roughly 30% less packet drop than MassDNS.

ZDNS is open source and can be found at Github.

This work was presented at IMC 2022 and awarded the community contribution award. You can read the paper and watch the conference talk.

Liz Izhikeivch (UC San Diego B.S. ’17, M.S. ’18) is a PhD Candidate/ NSF Fellow/Stanford Graduate Fellow at the Computer Science Department at Stanford University. Her research is at the intersection of networking, systems, and security.

Rate this article

Discuss on Hacker News

---

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.