DNS Shotgun is a benchmarking tool specifically developed for realistic performance testing of DNS resolvers. Its goal is to simulate real clients and their behaviour, including timing of queries and realistic connection management, which are areas where traditional tools are lacking.
Proper query timing is essential to simulate realistic cache-hit rate, while connection management becomes very important when benchmarking connection-oriented protocols.
In this blog post, we will focus on benchmarking DNS-over-TLS (DoT) and DNS-over-HTTPS (DoH) protocols. You can find more information about UDP, TCP, and DNS Shotgun’s features and methodology in this documentation.
Why build DNS Shotgun?
The rise in popularity of DoT and DoH presents us with new challenges. These connection-oriented protocols are built on top of TCP and TLS. The major difference from UDP is that they are no longer stateless. They require an established connection to exchange DNS messages. This introduces additional round-trips and performance overhead.
We wanted to measure this impact on both the clients and servers, at scale. What happens when thousands of clients switch from UDP to DoT/DoH? How does the client behaviour and protocol choice affect their latency and server performance cost? How much do you need to scale your infrastructure to handle the increased load of DoT/DoH?
To answer these questions, we needed a tool that would be capable of simulating real clients as realistically as possible.
Other existing DNS performance tools, such as dnsperf or flamethrower, establish a set number of connections at the start of the test and then send queries using these connections. Real clients, however, will keep establishing and closing their connections continuously.
The other issue with existing tools is the limited number of connections you can establish. DNS Shotgun has no such limit. It allows binding to multiple sources of IP addresses, using as many TCP connections as your hardware can handle. We’ve verified this can scale to at least hundreds of thousands of active established connections, illustrated in Figures 1 and 2.
What can DNS Shotgun do?
DNS Shotgun supports UDP, TCP, DoT and DoH. Traffic can be replayed over IPv4 or IPv6. The toolchain consists of various scripts for data processing, replaying traffic to simulate real clients and visualizing the results with charts such as the ones above.
The tool can be used to replay traffic over a single protocol or to use multiple protocols simultaneously. For example, it is possible to simulate the following scenario:
- 80% of clients use UDP
- 10% of clients use DoT
- 10% of clients use DoH
The results for each protocol can be interpreted separately and then compared to each other. You can find more information about DNS Shotgun, its features and usage in our comprehensive documentation. It describes the benchmark process step by step and also includes some performance tuning tips.
We’ve presented the results of our benchmarks of DoT and DoH at DNS OARC 33. Some of our UDP measurements were presented at RIPE 79.
A more general talk, with our recommendations for DoT/DoH deployments based on our benchmark experience, is also available in this video, which was originally presented at a DoH webinar organized by eco and CENTR.
DNS Shogun is built on top of the dnsjit engine, and wouldn’t be possible without the great work of DNS-OARC and Jerry Lundström for the development and continued support of dnsjit.
Tom Křížek is a free software developer who has contributed to various DNS software and utilities such as Knot Resolver or DNS Shotgun.
The authors would like to thank the Comcast Innovation Fund for sponsoring the work to support the use of TCP, DoT and DoH protocols.
This post was originally published on CZ.NIC blog.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.
registry.nic.cz/knot/shotgun:v20210714 seems to be a dead link (404)