The Domain Name System (DNS) is an essential protocol in the architecture of today’s Internet. It routinely translates domain names into IP addresses and also often handles a multitude of invalid queries. These include requests for non-existent domain names, called NXDOMAIN. A high volume of these invalid queries can adversely affect the online user experience, add strain to network operators, and in certain situations, might even pose a risk to the stability of DNS resolvers at different levels.
Qi Anx Xin (QAX) is a Chinese cybersecurity company and one of the major DNS infrastructure service providers in China. It operates the largest public DNS resolution system in the economy as well as the largest public PassiveDNS system. This gives us a broad and clear perspective, enabling nationwide observation of DNS performance and facilitating comprehensive analysis and interpretation of DNS operations.
This series will leverage the massive recursive resolver data from QAX’s public DNS to conduct an in-depth analysis starting on a particular dataset — NXDOMAIN data. The aim is to deeply understand the domestic DNS system’s operational state and thereby enhance the efficiency and security of the entire DNS resolution system.
The series will be divided into three parts:
- The first part compares the similarities and differences in erroneous domain name queries between China and the global DNS system and explores the reasons for these differences.
- The second part, from the perspective of domain names, comprehensively analyses the causes and impacts of erroneous domain names.
- The third part, from a regional perspective, compares the differences in error responses among different provinces in China and their causes. The goal is to more accurately understand and enhance the operational efficiency and security of the domestic DNS system.
NXDOMAIN (Non-Existent Domain, rcode=3) is a specific type of response in the DNS, indicating that the queried domain name does not exist in the DNS database. Among the various types of DNS error responses, NXDOMAIN is the most common, accounting for over 90% of all DNS error response types.
Comparison of root servers and recursive resolvers
Before starting the analysis, let’s look at the top server in the DNS hierarchy, the root server, in terms of NXDOMAIN responses. For illustration purposes, we will use the L-Root server as an example.
ICANN IHTI data
ICANN has analysed queries for the L-Root server to generate the Identifier Technology Health Indicators (ITHI) report. Figure 1 shows the L-Root server metrics from January 2018 to December 2023 from that report. The darker part of the graph indicates the percentage of NXDOMAIN data, which fluctuates between 50% and 80%. Geoff Huston analysed the reasons for the fluctuations in the root server’s NXDOMAIN ratio by comparing the trends of Chromoid on the APNIC Blog in July 2023. As of December 2023, the percentage of queries answered by NXDOMAIN was 55.27%.
| RANK | TLD | % OF ALL QUERIES |
|---|---|---|
| 1 | local | 7.292% |
| 2 | test | 1.944% |
| 3 | home | 1.163% |
| 4 | dhcp | 1.074% |
| 5 | lan | 0.999% |
| 6 | internal | 0.956% |
| 7 | ctc | 0.778% |
| 8 | bbrouter | 0.699% |
| 9 | arpa | 0.692% |
| 10 | localhost | 0.538% |
| 11 | localdomain | 0.522% |
| 12 | wifi | 0.475% |
| 13 | invalid | 0.336% |
| 14 | corp | 0.336% |
| 15 | jpg | 0.268% |
| 16 | svc | 0.238% |
| 17 | k8s | 0.215% |
| 18 | getcacheddhcpresultsforcurrentconfig | 0.205% |
| 19 | m4a | 0.200% |
| 20 | openstacklocal | 0.193% |
Recursive resolver data
Figure 2 shows the variation of the NXDOMAIN response rate by the QAX recursive resolver from July 2021 to December 2023, stabilizing at approximately 14% throughout 2023. It is roughly a quarter of L-Root’s. This difference primarily stems from their distinct roles and positions within the DNS — recursive resolvers are at the very bottom layer of the DNS, serving end-users directly, and often encounter repetitive queries.
These servers handle requests for various domain levels and optimize resolution speed by caching results. On the other hand, root servers are at the topmost layer, not oriented towards end users, with relatively fewer repetitive queries. They are mainly responsible for TLD queries, which frequently involve currently unresolvable TLDs, leading to a higher proportion of NXDOMAIN responses.
Table 2 shows a significantly diverse query distribution among the top 20 TLDs. The .arpa domain alone accounts for 30% (a large number of private IP PTR reverse queries, more on this later). In addition, some commonly used valid TLDs such as .com and .cn also appear in the list. This is because the NXDOMAIN responses from recursive resolvers are for non-existent domain names, not non-existent TLDs. .com has a large number of registrations, and .cn (China’s ccTLD) is associated with our data collection location.
| RANK | TLD | TLD TYPE | % OF ALL QUERIES | % OF NXDOMAIN QUERIES | % OF DE-DUPLICATED NXDOMAIN FQDNS |
|---|---|---|---|---|---|
| 1 | arpa | ICANN | 4.1667% | 30.3167% | 6.7615% |
| 2 | com | ICANN | 3.7407% | 27.2168% | 33.1975% |
| 3 | cn | ICANN | 0.9854% | 7.1698% | 4.2206% |
| 4 | net | ICANN | 0.7240% | 5.2681% | 4.7647% |
| 5 | org | ICANN | 0.6386% | 4.6464% | 2.0354% |
| 6 | ctc | 0.4406% | 3.2057% | 4.4355% | |
| 7 | lan | 0.2126% | 1.5470% | 7.8800% | |
| 8 | ru | ICANN | 0.1744% | 1.2691% | 0.1990% |
| 9 | wifi | 0.1185% | 0.8622% | 1.0409% | |
| 10 | cnp | 0.1115% | 0.8114% | 0.0000% | |
| 11 | localdomain | 0.0936% | 0.6811% | 0.1458% | |
| 12 | cc | ICANN | 0.0921% | 0.6701% | 0.1510% |
| 13 | local | 0.0820% | 0.5968% | 2.6110% | |
| 14 | xyz | ICANN | 0.0754% | 0.5487% | 0.6047% |
| 15 | top | ICANN | 0.0599% | 0.4359% | 0.3884% |
| 16 | dhcp | 0.0561% | 0.4082% | 0.3262% | |
| 17 | 3132372e302e302e31 | 0.0548% | 0.3984% | 0.0000% | |
| 18 | novalocal | 0.0545% | 0.3962% | 0.4084% | |
| 19 | rl=http | 0.0520% | 0.3783% | 0.0000% | |
| 20 | eu | ICANN | 0.0456% | 0.3317% | 0.0551% |
Responses from non-ICANN systems
To make analysis easier, we have divided the NXDOMAIN data into two categories based on the TLDs of the query names — queries that belong to the ICANN system and queries that do not.
Of the total NXDOMAIN data, the non-ICANN system (non-standard domain name queries) accounts for 19%. This is due to a variety of reasons, including system or network equipment configuration defects, application bugs, leftover data from test and sandbox environments, user errors and unauthorized tampering with inputs, as well as other unknown reasons leading to invalid access.
Attentive readers of this article will notice that some TLDs involve several major manufacturers. We have contacted some of these involved manufacturers through various channels, and they have all responded positively and made improvements. Specific names of the manufacturers are not listed.
| RANK | NON-ICANN TLD | % OF NON-ICANN NXDOMAIN QUERIES | DAILY QUERIES QUANTITY SCALE | DAILY FQDNS QUANTITY SCALE |
|---|---|---|---|---|
| 1 | ctc | 16.73% | 1B+ | 10M+ |
| 2 | lan | 8.07% | 1B+ | 10M+ |
| 3 | wifi | 4.50% | 100M+ | 1M+ |
| 4 | cnp | 4.23% | 100M+ | <1k |
| 5 | localdomain | 3.55% | 100M+ | 100k+ |
| 6 | local | 3.11% | 100M+ | 10M+ |
| 7 | dhcp | 2.13% | 100M+ | 1M+ |
| 8 | 3132372e302e302e31 | 2.08% | 100M+ | 1 |
| 9 | novalocal | 2.07% | 100M+ | 1M+ |
| 10 | rl=http | 1.97% | 100M+ | 1 |
| 11 | comp | 1.50% | 100M+ | <1k |
| 12 | openstacklocal | 1.46% | 100M+ | 1M+ |
| 13 | 0 | 1.38% | 100M+ | 10k+ |
| 14 | localhost | 1.33% | 100M+ | 100k+ |
| 15 | home | 1.13% | 100M+ | 1M+ |
| 16 | ***-wlan-controller | 1.00% | 100M+ | 1 |
| 17 | url | 0.74% | 100M+ | <1k |
| 18 | br-lan | 0.62% | 100M+ | 1 |
| 19 | bbrouter | 0.55% | 100M+ | 100k+ |
| 20 | null | 0.49% | 10M+ | 10k+ |
Compared to the L-root server, we noticed that many TLDs overlap in the two lists, but their rankings are significantly different. We selected a few TLDs and analysed the possible reasons for their appearance.
1. Reserved special-use domain names
.local and .localhost are officially reserved for special purposes. The .local domain is specifically reserved for multicast DNS (mDNS) environments. It’s used as a unique domain for hostnames in local area networks and can be resolved using the multicast DNS name resolution protocol. However, misconfigurations and other issues might cause these queries to be mistakenly sent to public DNS servers. The .localhost domain is reserved for loopback functions, where .localhost refers to the current computer accessing the domain.
2. Common LAN suffixes
.lan, .wifi, .localdomain, .dhcp, .home, and .bbrouter are typically used as default DNS suffixes for internal networks. Their use depends on specific network setups, device types, or organizational preferences.
3. Cloud environment suffixes
.openstacklocal and .novalocal are used as specific suffixes in cloud computing environments. .openstacklocal is used within private clouds or internal networks managed by the OpenStack cloud computing platform, identifying virtual machines and services. .novalocal is particularly associated with OpenStack’s Nova compute component and is used for domain names automatically assigned to virtual machines created on OpenStack.
4. Operator suffix
In the list above, .ctc has the highest percentage of non-ICANN TLDs. Statistics show that 92% of the client IP addresses accessing .ctc are from a specific operator in China. The default login address for the customized routers of this operator also ends with ctc, leading to the conclusion that the .ctc is primarily used by internal networks and services related to the devices of this operator.
5. Configuration errors
.cnp, .3132372e302e302e31 (which decodes to 127.0.0.1, the local loopback address), .***-wlan-controller, and .br-lan are examples of specific domain names that generate a lot of NXDOMAIN response queries due to network or application setup errors.
- .cnp
The cnp issue was caused by a misconfiguration of several subdomains of a manufacturer’s cloud service, where an extra ‘p’ was mistakenly added to the correct subdomains, resulting in a large number of NXDOMAIN responses.
| .CNP FQDN | % OF .CNP QUERIES | DAILY QUERIES QUANTITY |
|---|---|---|
| n-relay-ipc-txc-nj-00.***cloud.com.cnp | 62.58% | 100M+ |
| n-txc-relay-ipc-nj-01.***cloud.com.cnp | 24.13% | 100M+ |
| n-txc-relay-ipc-nj-00.***cloud.com.cnp | 12.23% | 10M+ |
| txc-transmit-ipc-nj.***clouds.com.cnp | 0.63% | 1M+ |
| n-txc-relay-tumscloud.***cloud.com.cnp | 0.40% | 1M+ |
| txc-relay-ipc.***cloud.com.cnp | 0.01% | 10k+ |
cnp issue.Latest update: The manufacturer has successfully replicated the issue in their environment and is actively engaged in diagnostic analysis.
- ***-wlan-controller
***-wlan-controller, which refers to a specific brand’s wireless LAN controller. From the DNS data, it is observed that this string is being queried extensively as a complete domain name, rather than as a TLD of other domain names.
In the configuration of routers from a specific brand, ***-wlan-controller is typically used as a domain prefix to aid wireless access points (APs) in dynamically discovering and connecting to the Wireless LAN Controller (AC) via the DNS. For instance, in the network environment of this router brand, if the DHCP server’s configured AC domain name is ac.example.com, an AP, upon receiving this domain name, automatically prefixes it with the ***-wlan-controller string, transforming it into ***-wlan-controller.ac.example.com for DNS resolution. This process helps the AP to determine the IP address of the AC. However, our DNS data indicates that this string is being queried extensively on its own, leading to a significant number of erroneous responses. This suggests the possibility of configuration errors or other issues, resulting in APs being unable to locate the correct AC, which could potentially affect network connectivity and stability.
6. Mobile application errors
.asia11, .com11, .cn11, .eu11, .asianull, .comnull, .cnnull, .eunull are primarily due to a large number of query requests generated by an error in a particular application on a specific brand of smartphone. A detailed analysis is as follows.
Our monitoring has revealed that TLDs such as .asia11, .com11, .asianull, and .comnull follow a similar pattern. They are all formed by combining common TLDs like asia and com with suffixes 11 or null. The response frequencies for these combinations are also very close, with requests including the 11 suffix accounting for 0.35% and those with the null suffix accounting for over 0.14%. The total number of domain requests related to these TLDs reaches up to several hundred million per day, involving client IP numbers in the tens of millions.
| RANK | NON-ICANN TLD | % NON-ICANN NXDOMAIN QUERIES | DAILY QUERIES QUANTITY SCALE | DAILY FQDNS QUANTITY SCALE |
|---|---|---|---|---|
| 25 | asia11 | 0.35% | 10M+ | <1k |
| 26 | com11 | 0.35% | 10M+ | <1k |
| 27 | cn11 | 0.35% | 10M+ | <1k |
| 28 | eu11 | 0.35% | 10M+ | <1k |
| 57 | comnull | 0.15% | 10M+ | 1k+ |
| 58 | cnnull | 0.14% | 10M+ | <1k |
| 59 | eunull | 0.14% | 10M+ | <1k |
| 61 | asianull | 0.14% | 10M+ | <1k |
This phenomenon is associated with a specific smartphone manufacturer. The global routing service domain names for this company are shown in Figure 5.
When (smartphone) devices from this manufacturer request these official domain names the correct domain names are erroneously appended with suffixes like ’11’ or ‘null’ due to code bugs. For example, the original request grs.***cloud.com is altered to grs.***cloud.com11. This leads to a large number of NXDOMAIN responses, resulting in abnormally high request volumes for TLDs such as .asia11, .com11, .asianull, and .comnull.
The graph below shows that requests for these incorrectly formatted domain names from this manufacturer began to appear as early as June 2022. Domains with the same error suffixes show consistent request frequencies. Domains ending with the ’11’ suffix ( grs.***cloud.asia11, grs.***cloud.cn11, grs.***cloud.com11, grs.***cloud.eu11) are represented in the chart by overlapping purple lines, and their request volumes surged starting from 22 July 2022. Similarly, domains ending with the ‘null’ suffix (grs.***cloud.asianull, grs.***cloud.cnnull, grs.***cloud.comnull, grs.***cloud.eunull) are depicted with overlapping blue lines, and their request volumes saw a significant increase beginning 17 November 2022.
We pinpointed the trigger for these types of queries through testing. Using this brand of smartphone, when swiping down from the top menu, domain name queries for the routing service with ’11’ or ‘null’ suffixes are sent out. The different suffixes are associated with its version.
Additionally, we have noted that the domain grs.***cloud.eu is already in an NXDOMAIN state, yet it still exists in the official support list. In either case, the massive number of this brand devices (smartphones) are generating invalid DNS queries. We recommend that this manufacturer update its system to correct this issue.
; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.13 <<>> grs.***cloud.eu
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 59655
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;grs.***cloud.eu. IN A
;; AUTHORITY SECTION:
***cloud.eu. 8 IN SOA monkey.dnspod.net. enterprise3dnsadmin.dnspod.com. 1700034101 3600 180 1209600 180
;; Query time: 21 msec
;; SERVER: 10.46.36.5#53(10.46.36.5)
;; WHEN: Wed Dec 13 17:55:59 CST 2023
;; MSG SIZE rcvd: 129Latest update: In response to this phenomenon, we provided feedback to the emergency response team of the company, and received a positive response. The conclusions of the analysis were communicated promptly. According to the latest feedback, the team has identified that the specific cause is due to an application configuration issue. Additionally, the domain status of grs.***cloud.eu was restored to normal shortly after the feedback was given.
Others
We also noticed a small number of queries with TLDs from the ‘Opennic,’ ‘Tor’, and ‘Namecoin’ namespaces. Generally speaking, these TLDs don’t leak into the public DNS resolution system. Their appearance here may be due to erroneous configurations (such as some test sandboxes) or incorrect usage (like non-Tor browsers attempting to connect to Tor domain names).
| TAG | TLD | % OF NON-ICANN NXDOMAIN QUERIES | DAILY QUERIES QUANTITY SCALE | DAILY FQDNS QUANTITY SCALE |
|---|---|---|---|---|
| OpenNIC | null | 0.4914% | 10M+ | 10k+ |
| Tor | onion | 0.0054% | 1M+ | 1k+ |
| OpenNIC | o | 0.0014% | 100k+ | 1k+ |
| Namecoin | bit | 0.0002% | 10k+ | 10+ |
| OpenNIC | oss | 0.0002% | 10k+ | 10+ |
| OpenNIC | pirate | 0.0001% | 10k+ | <10 |
| OpenNIC | geek | 0.0001% | 10k+ | 10+ |
| OpenNIC | libre | 0.0001% | 10k+ | 10+ |
| OpenNIC | gopher | 0.0001% | 10k+ | <10 |
| OpenNIC | bbs | 0.0000% | 1k+ | 10+ |
| OpenNIC | parody | 0.0000% | 1k+ | <10 |
| OpenNIC | dyn | 0.0000% | 1k+ | 10+ |
| OpenNIC | indy | 0.0000% | 1k+ | <10 |
| OpenNIC | chan | 0.0000% | <1k | 100+ |
| OpenNIC | oz | 0.0000% | <1k | 100+ |
| OpenNIC | cyb | 0.0000% | <1k | 10+ |
| OpenNIC | neo | 0.0000% | <1k | 10+ |
In the next part of this series, we’ll comprehensively analyse the causes and impacts of erroneous domain names from the perspective of domain names, before comparing regional differences in the final part.
Jinghua Bai, a Senior Security Analysis Engineer at Qi An Xin, specializes in DNS data analysis and big data mining.
Zaifeng Zhang contributed to this work.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.