Revisiting DNS and UDP truncation

By on 15 Jul 2024

Category: Tech matters

Tags: , , ,

Blog home

The choice of User Datagram Protocol (UDP) as the default transport for the DNS was not a completely unqualified success. On the positive side, UDP’s stateless query/response model has been a good fit for the stateless query/response model of DNS transactions between a client and a server. The use of a UDP transport enabled the implementation of highly efficient DNS server engines that managed high peak query rates.

On the other hand, these same minimal overheads imply that DNS over UDP cannot perform prompt detection of packet loss, and cannot efficiently defend itself against various approaches to tampering with the DNS, such as source address spoofing, payload alteration and third-party packet injection. Perhaps most importantly, the way UDP handles large payloads is a problem.

Payloads up to 65,507 octets can be loaded into a UDP frame when using an IPv4 transport. It’s 28 octets lower than the IPv4 maximum 16-bit packet length value due to allowing 20 octets for the IPv4 packet header and 8 octets for the UDP header. The maximum payload increases slightly to 65,527 octets when using an IPv6 transport (not using IPv6 Jumbogram extension headers), due to the observation that the 16-bit payload length field in the IPv6 packet header excludes the IPv6 packet header.

In practice, most networks do not cope at all with such large IP packets. IP fragmentation is used to adapt a large IP packet to be passed across a network path that uses a smaller maximum transmission size. That said, IP fragmentation is a problem.

RFC 8900, ‘IP Fragmentation Considered Fragile’ from September 2020 reiterates advice from a 1987 paper, ‘Fragmentation Considered Harmful‘, which points out that relying on the IP layer to perform the necessary adaptation to accommodate large payloads in a single datagram transaction is a very poor approach from the perspective of carriage performance.

Furthermore, a current work-in-progress in the IETF, ‘IP Fragmentation Avoidance in DNS over UDP‘, points out that fragmented DNS responses have systematic weaknesses that expose a DNS requestor to DNS cache poisoning from off-path attackers. As this work points out: “A DNS message receiver cannot trust fragmented UDP datagrams primarily due to the small amount of entropy provided by UDP port numbers and DNS message identifiers, each of which being only 16 bits in size, and both likely being in the first fragment of a packet if fragmentation occurs.


The DNS avoids IP fragmentation by restricting the maximum payload size carried over UDP. RFC 1035 contains the directive: “Messages carried by UDP are restricted to 512 octets (not counting the IP or UDP headers). Longer messages are truncated, and the TC bit is set in the header”.

The intent of setting the truncation (TC) bit in the DNS response was to indicate that the receiver should discard this UDP response and perform the same DNS query over a TCP transport. This is not exactly a highly efficient measure. The DNS query now takes an additional two Round-Trip Time (RTT) intervals (one for the DNS truncated response and a further exchange for the TCP handshake), and the server also needs to maintain a TCP state, which limits the server’s query processing capability. This option to re-query using TCP is preferably avoided but limiting DNS responses to, at most, 512 octets is not always feasible.

This limit of 512 octets becomes problematic in several scenarios. For example, a DNSSEC-signed query for the DNSSEC public keys of the root zone produces a response of 1,169 octets. It’s not just DNSSEC that is the issue here. We use the DNS for various forms of authentication, and it’s a common practice to load authentication codes into the DNS as TXT records. Multiple TXT records will all be packed into a response, which can lead to quite large responses. For example, a query for the TXT record for bbc.co.uk elicits a DNS response of 1,666 octets in size.

The workaround for this issue of a very conservative selection of the maximum UDP payload for the DNS was the use of a so-called pseudo-Resource Record, the OPT record. This is the general extension mechanism for DNS or EDNS. The specification for EDNS(0), RFC 6891, includes the option to use a UDP message size in the query to allow the querier to inform the responder of its capability to handle UDP responses greater than 512 octets, thereby avoiding some level of re-querying over TCP when the response is larger than this default size. RFC 6891 also contains the following advice: “Due to transaction overhead, it is not recommended to advertise an architectural limit as a maximum UDP payload size. … A good compromise may be the use of an EDNS maximum payload size of 4096 octets as a starting point.”

The IPv6 specification requires IPv6 networks and hosts to be capable of handling an IPv6 packet of up to 1,280 octets in size without fragmentation. The IPv4 specification has an unfragmented packet size of 68 octets, and IPv4 hosts are required to be capable of reassembling IP packets of up to 576 octets in length.

In practice, the original Ethernet packet sizes (64 to 1,500 octets) have been largely adopted by the Internet, and in most cases (where no encapsulation tunnels exist) packets of up to 1,500 octets will pass through the public Internet without triggering packet fragmentation. What this implies is that in proposing a UDP buffer size of 4,096 octets, then IP fragmentation of large DNS responses will be triggered, and all the issues raised relating to the use of UDP fragments may surface as a consequence.

If a primary objective is to avoid IP packet fragmentation, then a UDP buffer size of 4,096 octets is just too large. The advice in DNS Flag Day 2020 proposed the use of an EDNS(0) buffer size of 1,232 octets as a minimum safe size, based on the 1,280-octet unfragmented IPv6 packets, and making allowance for the IPv6 and UDP packet headers. However, this is a very conservative choice, and the downside is potentially unnecessary re-queries in TCP.

A current work in progress in the IETF, draft-dnsop-avoid-fragmentation, proposes that the EDNS buffer size should reflect not only the requestor’s maximum packet reassembly buffer size, but also the inbound network interface Maximum Transfer Unit (MTU), and where known, the network path MTU value. This working draft currently recommends a maximum buffer size of 1,400 octets.

Measuring EDNS buffer sizes

Which brings us to our first measurement question. To what extent do recursive resolvers follow this advice? What are the UDP buffer sizes used in queries from recursive resolvers to nameservers?

We have looked at the UDP buffer size in queries in June 2024, as shown in Table 1 and a cumulative distribution of this data is plotted in Figure 1. This is a query-weighted data set, using data from some 356,939,321 query sample points, collected over nine days from 25 June 2024 to 3 July 2024.

Table 1 shows the top 10 buffer sizes used in queries.

SizeQuery Count%
1,400197,925,50055.45%
4,09665,859,10418.45%
1,23257,497,61416.11%
1,22014,550,1914.08%
1,4527,296,6542.04%
5124,986,1221.40%
1,4102,343,9520.66%
1,4722,187,6060.61%
1,4401,495,6920.42%
1,680594,8260.17%
Table 1 — Top 10 UDP buffer sizes.
Figure 1 — Distribution of UDP buffer size values in DNS queries.
Figure 1 — Distribution of UDP buffer size values in DNS queries.

Just under 20% of these queries use a UDP buffer size greater than 1,472, which would appear to allow a responder to generate a fragmented UDP response unless, of course, it applies its own more stringent size restraints to the UDP response.

Measuring truncated responses

The next question is: ‘How effective is truncation in today’s DNS?’

Do DNS recursive resolvers always ignore the answer and additional sections of a DNS response if the truncated bit is set? And what proportion of resolvers can perform a DNS query over UDP in response to a truncated UDP response?

We’ve used the APNIC Labs ad-based measurement environment to perform this measurement,

This system uses online ads to enrol users to test particular behaviours from the perspective of the end user. The script in the ad performs several fetches of URLs. Each URL to be fetched uses a unique DNS name, and there is only a single authoritative DNS server to resolve this name, and a single web server to serve the named web object. We cannot instrument the end user browser that is running the ad script, but we can instrument the DNS and web servers to record the end of each measurement transaction.

Each fetch within a single ad script can measure specific behaviours or attributes, such as IPv6-only capability, DNSSEC for domain name validation use, QUIC protocol use, and the adoption of network behaviours that drop routes with invalid Route Origin Authorizations (ROAs). The only control framework of the script on the user side measures the elapsed time to perform each URL fetch and passes this information back to an ad controller by performing a final URL fetch with the individual time values encoded as attributes to this closing report.

The ad system is configured to present some 15M – 20M ad impressions per day, with an ad configuration that attempts to secure as wide a diversity of end users as possible. On the server side, we use a distributed network of five server-side platform clusters, located approximately on each continent to try and minimize the network delays when each user connects to the experiment’s DNS and web servers.

To perform this measurement of DNS resolver handling of truncated responses and the related ability to switch to use TCP we’ve used a glueless DNS technique. This allows us to use the DNS itself to detect whether a DNS resolution environment can resolve a DNS name that is constructed using a particular DNS behaviour.

Generically, the technique is to use a target DNS name that is itself delegated, and in the DNS delegation data, the glue records are deliberately omitted. This is shown in this example of glueless delegation.

example.com zone
child NS ns1.oob.example.net.

oob.example.net zone
ns1  IN   A   192.0.2.1

child.example.com zone
.    IN    A   203.0.113.1

In this example, to resolve the DNS name child.example.com, the recursive resolver will ask the nameserver for the example.com zone, and the server will respond with a referral record, indicating that the name is defined in the zone child.example.com, and the nameserver for that zone is ns1.oob.example.net. However, the referral response does not contain any glue records that provide an IP address for this name, so the recursive resolver must set aside its primary task (resolving child.example.com) and start a new task to resolve the name ns1.oob.example.net.

If it is successful, the resolver now has an IP address for the nameserver of the original target zone, and it can ask the final question. It will only ask this final query if the resolution of the nameserver name is successful.


In this case, we have modified the behaviour of the DNS server for the second zone (oob.example.net), such that all UDP responses to queries for nameserver names in this zone are truncated. We also use a categorization of these names such that one half of the experiment’s unique name set causes the nameserver to generate a truncated response (TC=1) with an empty answer section and the other half of the query names generates a perfectly normal complete DNS response with an intact and complete answer section, but with the TC bit set indicating (incorrectly in this case) that the UDP response has been truncated.

If the DNS resolver is using the contents of a truncated UDP response, then it will be able to obtain the IP address of the nameserver and make the third query without needing to re-query using TCP. A standards-compliant resolver will ignore the answer section of the UDP response that had the truncated bit set and re-query using TCP and use the TCP response to make the third query.

The behaviours are determined by performing a full recording of all packets that arrive at and leave our servers, and then analysing these packets to determine the DNS resolution query and response sequences for each individual experiment.

In a test conducted over June 2024, we found results as shown in Table 2

Tests 394,766,935
Ans+TC 197,173,501
No TCP 96,401
Rate0.05%
Table 2 — Incidence of use of answer section in truncated responses.

Across some 394M sample points, 197M tests were provided with a complete answer section as well as having the truncated bit set. Of these, 96,401 tests did not re-query using TCP, but performed the third target query, evidently using the contents of the answer section in the truncated response. The ten networks with the highest proportional use of truncated answers is shown in Table 3.

ASNCCSamplesFail rateAS nameEconomy
30549CA43160.56%LAKELAND-NETWORKSCanada
36923NG1,04554.55%SWIFTNG-ASNNigeria
17882MN127,49829.52%UNIVISIONMongolia
16509IN4,76123.61%AMAZON-02United States (India)
26421US21911.42%PONDEROSA-INTERNETUnited States
17816CN169,4889.42%China Unicom Guangdong ProvinceChina
42455IM1228.20%WI-MANX-ASIsle of Man
16284NG3344.79%Inq-DigitalNigeria
61272LT3284.27%ISTLithuania
16509DE4,8153.74%AMAZON-02United States (Germany)
Table 3 — Incidence of use of answer section in truncated responses (Top 10).

These results suggest that globally this aspect of DNS conformance to standards-specified behaviour is not a severe problem, and the incidence of the use of the answer section contained in truncated responses is just 0.05% of all samples.  However, as shown in Table 3, the incidence of this DNS resolver behaviour in specific networks is not so small, and this table lists the 10 networks with the highest incidence of the use of truncated responses where more than 100 samples were gathered over the month of June 2024.

Measuring TCP re-query

The second part of a resolver’s actions when receiving a DNS response over UDP that has the truncated bit set is to re-query using TCP. The related measurement question is: ‘What proportion of resolvers are incapable of performing a DNS query over TCP? ‘

The overall results are shown in Table 4.

There is a visible level of use of TCP-only here. 439,900 tests performed the DNS resolution by asking the initial query over TCP rather than UDP. This represents 0.11% of the total count of 394M tests. The remaining tests were initiated over UDP and given a truncated response.

Tests 394,766,935
TCP only 439,900
Rate0.11%
TC+UDP 394,327,035
UDP+NO TCP 10,555,279
Rate2.67%
            Table 4 — TCP use profile.

Of these 394M tests, 10.5M failed to re-query using TCP, or 2.67% of all such tests. The 10 largest TCP failure rates for networks with at least 500 sample points are shown in Table 5.

ASCCSamplesNo-TCP rateAS nameEconomy
9444HK12698.41%Hong Kong TelecommunicationHong Kong
22354TZ70296.72%University of Dar es SalaamTanzania
41937RS25,47595.43%MOJASUPERNOVASerbia
51357UA23994.98%SMARTCOMUkraine
37229TG8,08794.94%Atlantique TelecomTogo
10396PR339,52393.27%COQUI-NETPuerto Rico
16116IL89,85992.90%Pelephone CommunicationsIsrael
272744BR23991.63%DC INTERNET EIRELIBrazil
6535CL129,71091.00%Telmex ServiciosChile
38819HK110,12890.11%HKCSL GPRSHong Kong
Table 5 — Incidence of no TCP follow-up to truncated UDP responses (Top 10).

Once more, these results suggest that an Internet-wide behaviour to fail when performing a TCP re-query at a rate of a little under 3% of users is not a major issue for the DNS, yet there are individual networks, both large and small where there is a far higher TCP failure rate. Such high failure rates, more than 90%, of tests for users within each of these networks, suggest that the issue is likely to lie in the DNS resolver infrastructure operated by these networks rather than end clients performing their own DNS recursive resolver functions.

Conclusions

The DNS is attempting to steer a careful path between the issues associated with response loss associated with UDP and packet fragmentation and response loss with truncation and re-querying with TCP. Previous measurements of fragmented UDP failure rates in the DNS between recursive resolvers and authoritative nameservers showed a fragmented response failure rate of around 15%, while the current failure rate of truncation and TCP is far lower at some 3%. To improve robustness with large DNS responses, it appears that using a smaller maximum UDP size to avoid fragmentation and re-querying with TCP would be preferred due to lower potential loss rates.

This measurement exercise does not attempt to identify individual recursive resolvers. Modern high-capacity recursive resolver systems are compound systems composed of several DNS resolution engines. Some DNS resolver engines may use only UDP, where TCP tasks may be handed to other resolver engines that are specifically configured to manage the somewhat different TCP load profile. Without undertaking an effort to identify the modes of behaviour of these compound systems, identifying individual resolver systems by their IP address is not overly useful when trying to identify systemic behaviour anomalies.

A report on DNS re-query TCP failure rates can be found at APNIC Labs.

Rate this article

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.

Leave a Reply

Your email address will not be published. Required fields are marked *

Top