China’s NXDOMAIN data: Part 2 — Prominent domain patterns from QAX recursive resolver

As Part 1 of this series showed, there are various reasons for generating NXDOMAIN domain names. In this post, from the perspective of domain names, we’ll analyse the causes and impacts of erroneous domain names, but before proceeding with the analysis, let’s first look at one case.

An example case

In September 2022, Qi An Xin (QAX) received a lead from a root server operation team that found a large number of DNS queries in the form of MAC addresses from China on their root server. After checking with our monitoring data, we found that there were indeed a large number of such queries in the DNS logs.

Starting on 19 August 2022, these queries began flooding the DNS system, causing the daily volume of queries on the QAX recursive resolver to rise to nearly 2B. Further analysis revealed that this anomaly was caused by a feature flaw in a mobile app of a leading e-commerce platform, which inadvertently led users of the app to initiate DNS queries using the MAC addresses of their current network.

The vast user base of the app resulted in a massive influx of these abnormal queries to recursive resolvers and root servers. After identifying the cause, we reported the issue to the relevant departments. The bug was fixed on 18 September, and subsequently, these queries disappeared.

Process of association analysis

Through data association analysis, it was discovered that along with DNS requests for domains related to MAC, there are also DNS requests for dev, wlan0, lladdr, reachable. Additionally, these DNS requests were found to be associated with the domain names of a leading e-commerce platform.

By searching these keywords, it was found that these terms appear when checking the status of Wi-Fi connection hotspots. In Android, one can view the real-time status of connected Wi-Fi hotspots using the command ‘ip neigh show’.

Analysis with these unique strings found that almost all of the domain name requests were preceded or followed by domain names associated with a specific e-commerce platform, as illustrated in the provided diagram. After conducting tests and validations, it was confirmed that these requests were indeed related to this e-commerce platform. It appears that there was a programming flaw in the e-commerce platform’s app when processing the results of Wi-Fi hotspot connection checks. This flaw led to DNS queries being executed for all returned results (dev, wlan0, lladdr, MAC addresses), which in turn, caused the root servers to receive a large number of DNS query requests in the format of MAC addresses.

Domain name pattern analysis method

In the first part of our analysis, we focused on the proportion of NXDOMAIN responses and the distribution of TLDs, providing a macro perspective on the NXDOMAIN data. However, this analysis is limited to the highest level of the domain name structure and has not delved into more specific domain levels. As this top-level application error reveals, the large number of queries in the MAC address format highlights the need for a more detailed analysis at the domain name level. This type of analysis is critical to identifying and understanding the anomalous network behaviour triggered by specific patterns in domain names.

In fact, DNS recursive resolvers also cache the queried domain names that receive NXDOMAIN responses (RFC 2308). When dealing with large numbers of requests for a single domain, servers can make efficient use of the caching mechanism. Once the domain name resolution result is cached, subsequent queries for that domain can be responded to quickly, thus avoiding repeated resolutions. However, when faced with a large number of queries for many different domain names, the efficiency of caching decreases significantly as the server needs to frequently update and maintain caches for a wider range of domain names.

In light of this, a more in-depth analysis from the perspective of domain names follows, focusing on identifying significant domain name patterns in NXDOMAIN responses to classify and recognize similar domain name forms in the network. This will enable us to more comprehensively understand the characteristics of various domain names in the network and their underlying causes. Such detailed analysis is not only helpful in revealing and understanding abnormal network behaviour but also plays a significant role in optimizing DNS server performance and maintaining overall network security.

Generally, when a large number of NXDOMAIN abnormal responses occur, the corresponding queried domain names often exhibit specific patterns in their structure. Therefore, we will first generalize the Rname (FQDN) in NXDOMAIN responses to identify domain names with similar patterns, then we can further summarize and generalize these patterns and investigate the reasons for their occurrence.

Specific method:

1. Decomposing domain names into grams: All NXDOMAIN domain names are split into segments based on special characters (such as ‘.’, ‘_’, ‘*’, ‘&’, ‘-‘) and these segments are referred to as ‘grams’. For instance, djhsa.example-inc.com would be split into ‘djhsa’, ‘example’, ‘inc’, and ‘com’.
2. Statistical analysis and filtering of key grams: Count the frequency of occurrences for all grams. Then, based on a preset threshold, select the key grams, which are the keywords in NXDOMAIN domain names.
3. Generalizing domain names into patterns: Generalize each domain name by retaining its keywords and special characters. The other parts are generalized into patterns similar to ‘[a-z]{n}’ or ‘[0-9]{n}’ based on their character type and length. For example, djhsa.example-inc.com is generalized to [a-z]{5}.example-inc.com.
4. Aggregating, observing, and classifying patterns: Aggregate all generalized patterns, then observe and analyse the frequently occurring ones. Finally, based on the analysis results, refine rules based on the characteristics of these patterns.

Source analysis of prominent domain name patterns

For almost five months, we continuously monitored and analysed hundreds of millions of unique NXDOMAIN domain names collected daily by QAX’s recursive resolver. The analysis revealed that the reasons for generating large numbers of NXDOMAIN domain names fall into two main reasons — those caused by software applications and those caused by system configurations.

Problems caused by applications include chromoid, ads, user tracing, black grey app, blacklist service, and similar and problems caused by configuration include Device ID, Reverse DNS, FQDNs ending with search suffix, and so on. The specific distribution of each type is shown below.

During our monitoring period, we noticed that the form of domain names remained relatively stable. Based on this observation, we selected data from a specific date, 9 November 2023, for detailed analysis and illustration. To enhance the readability and visual appeal of the patterns, we replaced common types in the patterns, such as IPv4 addresses, IPv6 addresses, Universally Unique Identifiers (UUIDs), and hash values, with simplified identifiers like ‘IPV4’, ‘IPV6’, ‘UUID’, and ‘HASH’.

In Figure 5, the first layer represents all unique NXDOMAIN domain names in the QAX recursive resolver (de-duplicated based on FQDN). The second layer displays the proportions of various categories of domain names we have tagged. The third layer then details the specific domain name patterns for different categories and their respective proportions. Let’s attempt to analyse and explain the reasons behind each type and their potential impacts:

1. Chromoid

Chromoid refers to DNS queries with random letter strings between 7 to 15 characters in length, sent out by the Google Chrome browser to test for network abnormalities like DNS hijacking. After version 87, Chrome modified this detection method, and currently, these types of requests are reduced, accounting for 1.59% of the total. For detailed information about Chromoid domain measurements, please refer to this excellent article by APNIC.

2. Ads

Ads refer to subdomains related to advertising. The two main patterns are:

UUID-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.afdback.ptqy.gitv.tv

UUID-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.afdback.ppsimg.com

The term ‘afdback’ stands for advertisement feedback, indicating that these subdomains are related to transmitting advertising-related data. The two domains, gitv.tv and ppsimg.com, are associated with Internet television services. The presence of ‘ptqy’ points to a specific video application on smart Internet TVs. Upon analysis, it’s evident that these uniquely structured domain names are generated by the television application to acquire information about the DNS server currently used by the user.

As shown in Figure 6, this is the network traffic during the operation of the Android APK:

In the first returned JSON string, the ‘dns_url’ field contains a domain name in a special format, namely UUID-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.afdback.ppsimg.com, which is exactly what we observed in the DNS data. Each request generates a distinct UUID, bypassing the DNS cache, thereby enabling ppsimg.com’s authoritative server to receive all corresponding DNS requests. Consequently, this server can discern the DNS server details associated with each UUID, effectively identifying the source of each DNS request.

Subsequently, the client application sends an HTTP request with the same UUID (as shown in the second GET in the diagram above), allowing the web server to obtain the IP address corresponding to the user’s UUID. Using the UUID as a link, it’s possible to correlate the user’s external IP with the DNS IP they are using, as shown in the information from the second response.

{
  "version": "2020-08-19 v0.0.4",
  "ID": "15f765ac-ebdbd888433e-0abd1306-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "SN": "0abd1306",
  "USERIP": "35.221.56.251",
  "USER_AREA": "OVERSEA|US",
  "USERDNSIP": "74.125.18.2",
  "USERDNS_AREA": "GOOGLE |Public_DNS"
} 

The numerous one-time, non-existent domain names related to the television application’s advertisements observed on recursive resolver servers are not intended for actual network access. Instead, their purpose is to collect the IP addresses of the DNS servers currently used in the user’s environment through DNS queries. This is done to optimize the network performance for accessing their services based on the user’s location.

3. User tracing

User tracing refers to domain names associated with user tracking. The main patterns include UUID.clo.footprintdns.com and v17-[A-Z0-9]{8}.z.irs01.com, among others. These are primarily related to user visit footprints and online tracking. By querying these tracking-related subdomains, data such as users’ web browsing patterns, frequency of visits, and sources of access can be analysed. This information is used for behavioural analysis, targeted advertising, and other purposes.

4. Black grey app

The black grey app refers to domain names associated with the black and grey industry. The number of domain names and client sources in this category show a high degree of consistency in their patterns.

If directly accessing certain second-level domain names within these patterns, such as 608.vip, it automatically redirects to the download page of a gambling application. The emergence of so many NXDOMAIN subdomains with similar patterns suggests that it might be a case of industry sabotage, using random subdomains to attack the domain names of competitors.

5. Blacklist service

Blacklist service refers to subdomains related to blacklist services. From our patterns, we identified various filters used by different service providers for screening spam content. These services include:

• DNS-based Blackhole List (DNSBL) services, such as IPV4.bl.spamcop.net and IPV4.zen.spamhaus.org, which performs reverse DNS queries on sender IP addresses to determine if they are listed on spam sender blacklists.
• Uniform Resource Identifier Blacklist (URIBL) services, like [a-z0-9]{8}.uribl.spameatingmonkey.net, blocks spam based on email content (typically domain names or websites).
• Hash-based Blackhole List (HASHBL) services, such as [a-z0-9]{8}.ebl.msbl.org, which stores blocklist data as one-way encrypted hashes to decide whether to accept, filter, or reject incoming emails.

These patterns’ domain names are specifically used for identifying and blocking spam emails, phishing, and other malicious online activities. Each domain represents a specific service or database, which checks if sender IP addresses, mail servers, website URIs, and so on, are marked or have adverse records in these blacklists or threat intelligence sources. This helps to assess the credibility and security threat level of senders, websites, or links, thereby filtering out spam content and preventing attacks.

6. Device ID

Device ID refers to subdomains related to the device ID, with the primary pattern being [0-9]{10}.HASH.device.hiwifi.com. ‘HIWIFI’ is a wireless router, and these types of subdomains may be used to differentiate and manage devices connected to this model of wireless router. They could also be used for device identity verification or communication for specific purposes.

7. Reverse DNS

Reverse DNS refers to subdomains used for reverse domain name resolution, with patterns such as IPV4.in-addr.arpa and IPV6.ip6.arpa. Reverse DNS resolution is a special DNS service that uses PTR records (pointer records) to convert IP addresses back to their corresponding domain names. Simply put, it does the opposite of standard DNS resolution — where standard DNS translates domain names into IP addresses, reverse DNS identifies the corresponding domain names from the IP addresses.

8. Reserved name: .local

Reserved name: .local represents all domain names with the Top-Level Domain (TLD) name of .local. The domain name local is a special domain reserved for multicast DNS environments, which is different from the usage of public DNS (unicast DNS). We categorize it separately to conduct an in-depth analysis focusing on its domain name patterns within the public DNS, as well as analysing the reasons for its leakage into the public DNS.

Figure 9 provides a detailed representation of the distribution of all domain names with .local as the TLD in the QAX recursive resolver. The first layer in the diagram shows all domain names with .local as the TLD. The second layer displays the proportion of different patterns within these domain names. The third layer then presents the ratio of sub-patterns, along with an example domain name of each sub-pattern.

From the above chart, it is evident that among all domain names with .local as the TLD, UUID.local accounts for 94%. This indicates that the majority of domain names with the .local suffix follow the UUID.local format.

The following chart illustrates the development trend of the UUID.local pattern domain names. Although we cannot pinpoint the exact time of their initial appearance, the change in their quantity over time can be observed through our public DNS data.

In Figure 10, the red bar graph represents the number of requests, reaching the scale of hundreds of millions per day, corresponding to the vertical axis on the left side of the chart. The green bar graph shows the number of domain names following this pattern, with a scale in the tens of millions per day, also corresponding to the left vertical axis. Meanwhile, the blue curve represents the number of clients initiating requests, with the scale in the millions per day, corresponding to the vertical axis on the right side of the chart.

Upon deeper analysis, it was found that the UUID.local pattern emerges as part of a solution to prevent privacy leaks in WebRTC. The widespread occurrence of this pattern on the Internet may be associated with a bug in a certain open-source component or compatibility issues with different versions of WebRTC.

The emergence of UUID.local

WebRTC enables web browsers and mobile applications to add real-time audio and video peer-to-peer connections and communication capabilities. This technology allows users to access platforms like Bilibili, Douyin, and Kuaishou directly within browsers and mobile apps, using its peer-to-peer connection features for interactive activities such as live streaming connections. However, WebRTC carries the risk of privacy leakage, as it can expose the user’s real IP addresses, both public and private.

To address the issue of private IP leakage, there is an IETF draft proposal. Specifically, it involves hiding the IP address using dynamically generated multicast DNS (mDNS) names. When this solution is enabled, during real-time audio and video peer-to-peer connections and communication with WebRTC, it broadcasts its mDNS name, which is UUID.local, within the local network. The UUID represents a randomly generated unique identifier.

The destination address for mDNS requests is a fixed multicast address, 224.0.0.251. Normally, this type of traffic should not leak into the public DNS environment.

Analysis of UUID.local leakage

To further pinpoint and analyse the reasons behind the large volume of UUID.local occurrences in the public recursive network, we set up our own DNS traffic analysis system in the local network. This system is capable of analysing DNS traffic from various network devices such as smartphones and PCs in a multi-dimensional manner.

Considering the application scenarios of WebRTC, we hypothesized that it might be related to video websites or the currently popular live-streaming platforms. After testing several popular video applications, including Bilibili, Kuaishou, and Douyin, we discovered that the live streaming module of one of these applications, specifically the Android version, triggers DNS requests for UUID.local.

In Figure 12, we can see that the domain names in the requests occurring before the DNS request  UUID.local are all related to this application. Particularly, the parts related to STUN indicate the use of a protocol for NAT traversal to establish peer-to-peer communication, which is a crucial technology supporting the WebRTC protocol.

Combining DNS traffic analysis with the analysis of the app, we speculate that there are three main possible reasons:

1. Configuration error: When using WebRTC in the live streaming module, this Android app does not correctly construct multicast DNS request addresses (224.0.0.251) and continues to use regular DNS addresses, leading to leakage onto the public Internet.
2. Open source component error: This Android app’s live streaming module uses an open-source implementation of WebRTC, which might have certain vulnerabilities, leading to situations where multicast DNS requests are leaked onto the public Internet.
3. Compatibility issue: Differences in WebRTC versions between the communicating parties could result in mDNS being announced to the broader network in a unicast form.

It is currently not possible to further pinpoint the specific reason. However, based on the distribution pattern of .local, the latter two possibilities seem more likely. After all, other domain names ending with .local are relatively rare on the public DNS, aside from the UUID.local format. Additionally, data published by the L-Root server shows that .local accounts for the largest proportion of its erroneous responses. In the case of error requests on 1.1.1.1, .local also has the highest share, which may imply that a similar issue exists in global WebRTC usage.

Latest update: We have established contact with the security team of the video application in question. In response to the issues we raised, they conducted an in-depth analysis and have resolved the problem in the latest version of their app. However, it is worth noting that the observed volume of UUID.local queries on the larger Internet has not significantly decreased. This suggests that the issue is not confined to a single company but is more likely a widespread issue caused by a commonly used open-source component.

9. FQDNs end with search suffix

FQDNs that end with a search suffix are domain names appended with a search domain suffix. When a user-queried domain name does not exist in the DNS, the system automatically tries to append the configured search domain suffix to the domain name and initiates another DNS request. The settings for these search suffixes are typically associated with specific network environments, operating systems, or device types. Structurally, there are no specific restrictions on the search domain suffix; it can be any level of domain name, including TLDs, second-level domains, third-level domains, and so on.

Number	SUFFIX
1	smartont.net
2	ctc
3	wifi
4	lan
5	dhcp
6	home
7	huawei.com
8	router.ctc
9	airdream
10	bbrouter
11	tendawifi.com
12	realtek
13	openstacklocal
14	mshome.net
15	wowifi.smartont.net

Looking at the top five search domain suffixes, in Figure 14, the second layer shows the proportion of domain names for each search domain, the third layer displays the subcategories within each domain, and the fourth layer shows their specific domain name patterns.

We can observe that:

1. Figure 14 shows the detailed distribution of five search domain suffixes where .lan is a commonly used default suffix in local area networks, .smartont.net and .ctc are related to network operators, .wifi is frequently seen in Wi-Fi devices, and .dhcp is common in dynamically allocated network environments.
2. In different search domains, many domain name patterns exhibit similarities. For instance, Chromoid(.suffix) represents Chromoid domain names with a specific search domain suffix. A similar pattern is also observed in other domain names like Ads(.suffix) and UUID.local(.suffix). In this pattern, domain names from the same source form a unique structure by adding their respective specific search domain suffixes. This phenomenon indicates that certain types of domain name patterns are commonly found across search domains.
3. Some domain name patterns appear exclusively within specific search domains. For example, Device ID(.suffix) only appears in the .lan search domain. This might reflect an intrinsic association between these domain names and specific search domains. Specifically, the presence of Device ID(.suffix) solely within the .lan search domain suggests that the default search domain suffix for this router (hiwifi.com) might be .lan.

10. Others

Table 2 lists the patterns that are not included in the previous nine category patterns. These unidentified patterns represent the focus of our subsequent attention.

PATTERN	DAILY FQDNS QUANTITY SCALE	FQDN EXAMPLE
[a-z0-9]{8}	1M+	e23c753e
[a-z0-9]{8}.test	1M+	001eab65.test
[a-zA-Z0-9]{17}.mdbook.cn	1M+	50d3d5c8fdb0d5a70f1c4c65d.mdbook.cn
[a-z0-9]{4}.kaiyun.com	1M+	6u3b.kaiyun.com
&ver=[0-9]{1}&id=[A-Z0-9]{11}	1M+	&ver=1&id=1782071404875474966
[a-z0-9]{24}.jpg	100k+	gkwrimaiw418aaaknwjr6oxv.jpg
[a-z0-9]{4}.baidu.com	100k+	rjd6.baidu.com
[a-z0-9]{4}.spd-inv.net	100k+	kd0i.spd-inv.net
[a-zA-Z0-9]{16}.mdbook.cn	100k+	a0ea7053ca2143937c3d4c65.mdbook.cn
[a-z0-9]{4}.baidu2.com	100k+	s4ef.baidu2.com
[a-z0-9]{4}.baidu3.com	100k+	f8lt.baidu3.com
[a-z]{4}.kaiyun.com	100k+	fnud.kaiyun.com
IPV4	100k+	192-168-201-167
inst-[a-z0-9]{29}-[a-z0-9]{16}	100k+	inst-zef83nzhzf96vbz5eccmccnnapkvm-b6v3ytbfqjvq32td
[a-z]{4}.baidu.com	100k+	lfjr.baidu.com
[a-z0-9]{9}.dyn.telefonica.de	100k+	x4db29bb2.dyn.telefonica.de
[a-z]{4}.spd-inv.net	100k+	iltg.spd-inv.net
[0-9]{7}.b2b.jinanfa.cn	100k+	2233615.b2b.jinanfa.cn
[a-z]{4}.baidu2.com	100k+	lxwi.baidu2.com
[a-z0-9]{11}	100k+	47636zdl709

Automated monitoring

To achieve automated tracking of NXDOMAIN response data, we monitor error response data (covering the main types of error codes) and have implemented monitoring of various types of errors and anomalies using the method introduced earlier, specifically:

1. The NXDOMAIN response data is monitored by dividing the corresponding query domain names into identified patterns and unidentified patterns. Identified patterns refer to domain name patterns that we have observed in historical data and have tagged with specific labels. Unidentified patterns, on the other hand, are either newly emerged or previously unnoticed (generally, patterns with smaller volumes are not noticed).
2. Whether it’s identified or unidentified patterns, any significant increase in the error response data they represent is worth analysing to determine the cause. This is especially true for unidentified patterns, where it’s crucial to delve deeper to see if the increase is due to a surge in requests for a newly emerged pattern category. Subsequently, situations where the cause can be determined should be addressed with alerts and appropriate actions.

Using this method, we can achieve timely detection of errors in DNS data, such as incidents where an e-commerce platform inadvertently sends out user MAC addresses.

In the final part of this series, we’ll take a regional approach and compare the variations in error responses among various provinces in China and their root causes to gain a more precise understanding of the domestic DNS system’s operational efficiency and security.

Jinghua Bai, a Senior Security Analysis Engineer at Qi An Xin, specializes in DNS data analysis and big data mining.

Zaifeng Zhang contributed to this work.

Rate this article

---

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.