RFC 9199 is an Internet Engineering Task Force (IETF) informational document that presents considerations for large authoritative DNS server operators.
These considerations are derived from six peer-reviewed academic papers (published between 2016 and 2021) that my RFC coauthor and I at SIDN Labs and the University of Southern California/Information Sciences Institute have referenced, and which we at SIDN have used to improve the security, resilience, and performance of the .nl ccTLD.
This post explains the process my fellow authors and I followed to develop this document and provides a summary of the six considerations.
First, what RFC 9199 is not
Before we explain the considerations, it’s important to specify that RFC 9199 is not an IETF consensus document or a protocol standard. It is an informational document — its sole purpose is to document our findings.
Even though most people think of RFCs as protocol standard documents, that is only one category of RFC. RFCs can belong to multiple categories:
- Informational (such as RFC 9199)
- Standard (such as RFC 4033, which covers DNSSEC)
- Best current practices (such as RFC 9210, which covers DNS transport over TCP and operational requirements)
- Experimental (such as RFC 9057, which covers the e-mail author header field)
- Historic
Each category has its own purpose. The drafts can come from various sources, referred to in the IETF context as streams:
- IETF: Typically from a Working Group (such as DNSOP, which standardizes protocols)
- Independent: Any individual can submit a draft document outside the IETF stream
- IAB: Documents published by the Internet Architecture Board (IAB)
- IRTF: Documents published by the Internet Research Task Force (IRTF)
From document draft to RFC
My coauthors and I submitted our draft document to the IETF DNSOP Working Group (IETF stream) in March 2019. While feedback was neutral to positive, the Working Group rightfully concluded they could not contribute to improving the document, given the conclusions were derived from already peer-reviewed and published research works.
An IETF Working Group is a suitable stream to use when designing a new protocol, as we did previously with RFC 6781 (DNSSEC Operational Practices). However, we found it was not the right stream when presenting conclusions that have already been reached. We therefore changed the stream of our draft from IETF to Independent. That implied a different evaluation path, on which the Independent Submission Editor chooses reviewers for the document, just as an academic journal editor chooses reviewers for an article. That is very different from the IETF stream, in which anyone on the IETF can comment on the draft.
After individual reviews and multiple iterations, our document was published as an Independent Submission with Information status.
So, what is RFC 9199?
The RFC presents our considerations (advice about what to do) upfront and refers the reader to the respective papers for details about what we have determined. The added value of RFC 9199 is that it summarizes our research conclusions and frees the operator to use our findings without reading through multiple dense academic papers.
Consideration #1: Deploy anycast on every authoritative server
In 2015, many authoritative server operators, such as the Root Zone operators and .nl (SIDN), used a mixed setup of authoritative servers; some were unicast, others were anycast. Anycast adoption was still progressing at the time.
We have shown that if you are a large global provider, it is in your best interest to use anycast on every authoritative server for your zones, for two reasons:
- Anycast servers cope better with DDoS attacks than unicast counterparts.
- Anycast servers deliver far faster responses than unicast servers because clients can be served by nearby servers rather than by one server at a single location. And resolvers will see all available authoritative servers — so unicast servers will impact them too.
Fast-forward to 2022, all root server operators have deployed IP anycast. Anycast adoption from 2017 to 2021 has also grown for ccTLDs and other zones. We cannot speak for all the root operators, but SIDN’s migration of .nl to all-anycast services was driven partly by our research.
Consideration #2: Routing matters more than the number of sites for anycast
When comparing anycast DNS providers, one common metric is the number of global anycast locations each provider has. Typical anycast networks range from fewer than ten sites to more than 150 (L-Root, for example).
Intuitively, one might think that more instances are better and will always lead to shorter response times. However, that is not necessarily true. Proper route engineering can matter more than the total number of locations.
We showed that an anycast network with eight locations could deliver similar latency to clients as the K-Root and L-Root were then delivering, with 33 and 144 locations at the time of the study. So, routing optimization can be more important than the number of global locations, depending on your client population and query distribution. We also showed that poor routing can lead to anycast polarization, which causes large clients to see an anycast service as unicast (they reach only one location).
We therefore recommend that operators monitor the latency of their clients, which can also be done using passive TCP queries.
Consideration #3: Compile anycast catchment maps to improve design
Suppose you have an operational anycast network and want to expand it by adding new locations. Now, depending on your peering/transit providers and your client population, adding a single location can lead to massive traffic changes — sites that had lots of traffic may end up receiving little. It is very hard to predict such traffic changes.
To avoid unwanted surprises, operators can measure what effect a proposed change will have on an adjacent network. To measure the effect, the operator needs to announce a test prefix on every location operated on the production network. They can then use these prefixes to establish where the IPv4 client population will be mapped to, using a tool we have developed called Verfploeter. Verfploeter sends ICMP echo requests (ping packets) to an IPv4 hit list, and the responses go to individual anycast locations. The distribution of the responses received at a given location is that location’s ‘catchment’.
The impact of each new routing change can be measured by running Verfploeter repeatedly. One limitation of Verfploeter is that it does not currently support IPv6, as the IPv4 hit lists used are generated by means of frequent large-scale ICMP echo scanning, which is not possible using IPv6.
Consideration # 4: Employ two strategies when under stress
DDoS attacks are still a significant threat to DNS operators. How should a DNS operator engineer their anycast authoritative DNS server to respond to a DDoS attack?
We found out empirically that there are two main strategies for defending against a DDoS attack:
- An operator can withdraw their routes, pre-prepend their AS route to some or all their neighbors, perform other traffic-shifting tricks (such as reducing route announcement propagation using BGP communities), or communicate with their upstream network providers to apply filtering (potentially using FlowSpec) or the DDoS Open Threat Signalling (DOTS) protocol (RFC 8811, RFC 9132, RFC 8783). Such techniques shift legitimate and attack traffic to other anycast instances (hopefully with greater capacity) or block traffic entirely.
- Alternatively, operators can let sites become degraded absorbers by continuing to operate them, knowingly dropping incoming legitimate requests due to queue overflow. However, that approach will also absorb attack traffic directed toward the catchment, hopefully protecting the other anycast instances.
Consideration # 5: Consider longer time-to-live values whenever possible
Caching is the cornerstone of good DNS performance and reliability. A 50ms response to a new DNS query may be considered fast, but a response of less than 1ms to a cached entry is far quicker. We have shown that caching also protects users from short outages and even significant DDoS attacks.
On DNS resolvers, authoritative DNS server operators directly control caching (analogous to short-term memory). This is because each DNS record has a time-to-live (TTL) field that specifies how long a record should remain in the resolver’s cache.
We recommend that operators use TTLs of at least four hours (possibly more) for their records because it makes responses faster. Exceptions should be made for load balancers and some DDoS-based solutions that require short TTLs (although 15 minutes may provide sufficient agility for many operators).
Consideration # 6: Consider the difference in parent and child TTL values
In the DNS, there is some level of information replication, both on parent and child authoritative servers. For example, the NS records of the domain example.com can be retrieved from the parent .com authoritative servers:
dig ns example.com @e.gtld-servers.net
;; AUTHORITY SECTION:
example.com. 172800 IN NS a.iana-servers.net.
example.com. 172800 IN NS b.iana-servers.net.But they can also be retrieved from the child authoritative servers:
dig NS example.com @a.iana-servers.net.
; ANSWER SECTION:
example.com. 86400 IN NS a.iana-servers.net.
example.com. 86400 IN NS b.iana-servers.net.Look at the TTL fields — the parent server reports a 172800 second TTL, while the child reports an 86400 second TTL. Which one should the resolver trust?
We found that 90% of resolvers trust the child’s authoritative response. The important conclusion from our study is that authoritative operators cannot depend on their published TTL values alone — the parent’s values are also used for timing cache entries in the wild. Operators planning infrastructure changes should assume that an older infrastructure must be left on and operational for at least the longest of the two TTLs.
Researchers and RFCs
Overall, our experience with the RFC process has been very positive. Although it took a long time to achieve RFC status (three years and four months from the first draft to the RFC publication), it allowed us to reach an audience (operators) that would have been unaware of our findings if they had remained confined to academia.
Communicating the findings to a wider audience is mutually advantageous. The operators benefit from information they can put to practical use, and we, the researchers, from the feedback and comments operators provide in various phases of the process. We are grateful for their patience and help in the process.
Joint effort
As mentioned, RFC 9199 is a summary of the primary considerations of six research papers written over six years. The authors of those papers and the following people who contributed substantially to the content should therefore be considered co-authors. This document would not have been possible without their work:
- Ricardo de O. Schmidt
- Wouter B. de Vries
- Moritz Müller
- Lan Wei
- Cristian Hesselman
- Jan Harm Kuipers
- Pieter-Tjerk de Boer
- Aiko Pras
Adapted from the original post which appeared on the SIDN Labs Blog.
Giovane Moura is a Data Scientist with SIDN Labs (research arm of SIDN, the .nl registry) and Guest Researcher at TU Delft, in the Netherlands.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.