Taking a closer look at the NTP pool using DNS data

Established in 2003, the NTP pool is a voluntary project comprising dynamic virtual clusters of networked computers that provide highly accurate time to millions of systems around the world — it’s the default time server for most of the major Linux distributions and many networked appliances.

The computers that are ‘in the pool’ are part of the pool.ntp.org domain and are divided into multiple subdomains based on geography and specific ground-rules. DNS polling is used on these subdomains to provide the required server IP address for the client to use. This technique is actually somewhat similar to the Fastflux technology of DNS used by some malware.

For someone who does research on DNS Fastflux detection, pool.ntp.org is certainly no stranger.

Our DNSMon system uses a small number of domain names in the NTP pool due to the huge amount of access requests, as well as a small number of DNS names mapping to a large number of scattered IP addresses.

From the DNS perspective, in addition to being able to assess the scale of the NTP pool itself, it can also be used to measure some basic Internet services. After all, almost all networked devices must be time-synchronized (we do see a small number of devices having problems synchronizing time but that is less than 0.02%).

The following analysis is based on 24-hour NTP pool data we gathered from 19 May 2020 18:00 (CST) to 20 May 2020 18:00 (CST) from DNSMon.

Users

On the official NTP pool website, it shows that the number of users ranges from 5 million to 15 million, with around 4,000 server IPs.

During the 24-hour period of our study, a total of 880,000 different clients accessed the time synchronization service provided by the NTP pool. Considering the data coverage of our system, the number of users who use the NTP pool time synchronization service in China is quite near the upper limit of the number of users declared on their official website.

Server distribution

The number of NTP pool server IPs is 3,758 (1,028 IPv6 and 2,730 IPv4) distributed across 97 economies and regions around the world. They are mainly concentrated in economies with well developed networks, such as the United States, Germany, France, Britain, and the Netherlands. Together, these five economies account for 55% of NTP servers.

This study examined NTP servers in mainland China, Hong Kong, Taiwan and Macao, and found the number of NTP server IPs in these economies accounted for only 2% of the total number of active IPs in the total NTP pool. This value is much lower than the proportion of other network services. There are 25 operators in this NTP pool:

      7 3462|Data_Communication_Business_Group
      6 37963|Hangzhou_Alibaba_Advertising_Co.,Ltd.
      5 4538|China_Education_and_Research_Network_Center
      5 45090|Shenzhen_Tencent_Computer_Systems_Company_Limited
      4 132203|Tencent_Building,_Kejizhongyi_Avenue
      3 4808|China_Unicom_Beijing_Province_Network
      2 9381|HKBN_Enterprise_Solutions_HK_Limited
      2 9304|HGC_Global_Communications_Limited
      2 36351|SoftLayer_Technologies_Inc.
      2 133752|Leaseweb_Asia_Pacific_pte._ltd.
      2 10229|Internet_Content_Provider
      1 9312|xTom
      1 8075|Microsoft_Corporation
      1 58461|No.288,Fu-chun_Road
      1 55990|Huawei_Cloud_Service_data_center
      1 5580|Hibernia_Networks_(Netherlands)_BV
      1 4847|China_Networks_Inter-Exchange
      1 4780|Digital_United_Inc.
      1 4609|Companhia_de_Telecomunicacoes_de_Macau_SARL
      1 45102|Alibaba_(US)_Technology_Co.,_Ltd.
      1 4134|No.31,Jin-rong_Street
      1 23734|Netrouting_Inc
      1 17964|Beijing_Dian-Xin-Tong_Network_Technologies_Co.,_Ltd.
      1 139240|Starch_Works
      1 131584|Taiwan_Intelligent_Fiber_Optic_Network_Co.,Ltd. 

From a geographical point of view of this study, the servers participating in the NTP pool service are in mainland China (24), Hong Kong (18), Taiwan (11) and Macau (1).

NTP pool subdomain distribution

At present, the categories of subdomains are mainly divided in three ways:

• Continent
• Economy
• Vendor

Among them, the continents and economies are divided based on the concept of geographical zoning. The core idea is similar to the EDNS Client Subnet (ECS) in the DNS — trying to provide an NTP server that is close to the geographical location of the user’s source.

The NTP pool for vendors provides subdomain names with a high degree of identification for specific vendors (router vendors, operating systems, and other hardware and software vendors). Vendors can use the subdomains provided by the NTP pool directly within their products. For example:

 0.vendor.pool.ntp.org
 1.vendor.pool.ntp.org
 2.vendor.pool.ntp.org
 3.vendor.pool.ntp.org

Visit the NTP pool vendor page for more details.

According to our statistics, within the 24-hour period of our study, 682 NTP pool domain names showed up in our DNSMon, including 534 valid subdomains and 148 invalid subdomains (see the next section). The subdomain names on top of the list below are, as expected, mainly based on economies and regions such as cn/hk/tw/jp/sg and visits based on Asia. The other perspective is based on the access of operating systems such as android/openwrt/centos, and the access of native [0-3] .pool.ntp.org.

  146677 "cn.pool.ntp.org"
  145710 "asia.pool.ntp.org"
  143637 "2.android.pool.ntp.org"
  109730 "1.cn.pool.ntp.org"
  109123 "hk.pool.ntp.org"
  108859 "tw.pool.ntp.org"
  107648 "jp.pool.ntp.org"
  107471 "sg.pool.ntp.org"
   93682 "2.asia.pool.ntp.org"
   91415 "0.pool.ntp.org"
   82659 "pool.ntp.org"
   81139 "0.cn.pool.ntp.org"
   77800 "0.asia.pool.ntp.org"
   77077 "2.pool.ntp.org"
   73512 "3.cn.pool.ntp.org"
   72855 "1.asia.pool.ntp.org"
   71965 "2.openwrt.pool.ntp.org"
   71907 "3.pool.ntp.org"
   70814 "1.pool.ntp.org"
   70158 "0.centos.pool.ntp.org"

Differences in access by different vendors

Among the top subdomains, we see vendor names such as android/openwrt/centos. We examined our data and generated the following word cloud according to the number of visits:

The vendors include not only the common Linux releases, but also some network equipment manufacturers, as well as some consumer smart devices and security network products. From the security perspective, some users probably don’t really want their device types to be leaked this way.

Nearly 3% of the DNS request domain names of the NTP pool are invalid

Further, we found that nearly 3% of the DNS request domain names of the NTP pool are invalid. There are 148 invalid domain names (except ap.pool.ntp. org), of which the NTP time synchronization service was provided but later stopped. None of the remaining domain names have provided NTP time synchronization services. These included:

• Old devices that have not been updated so they still use old NTP pool domains such as the above mentioned domain name: ap.pool.ntp.org.
• The initial built-in NTP pool domain name has a typo, such as asis.pool.ntp.org or asian.pool.ntp.org.
• Buggy NTP clients causes the wrong domain name to be requested, for example, the wrong prefix “www” was accidentally added to www.africa.pool.ntp.org, www.europe.pool.ntp.org and www.oceania.pool.ntp.org.

The top 50 domain names and the number of requests (accounting for 98.26% of the total number of invalid requests) are as follows:

   18468 "2.generic.pool.ntp.org"
   18423 "1.generic.pool.ntp.org"
   18407 "0.generic.pool.ntp.org"
   18374 "3.generic.pool.ntp.org"
    3676 "4.pool.ntp.org"
    1538 "www.2.android.pool.ntp.org"
    1372 "www.africa.pool.ntp.org"
    1360 "www.europe.pool.ntp.org"
    1357 "www.south-america.pool.ntp.org"
    1339 "www.asia.pool.ntp.org"
    1331 "4.asia.pool.ntp.org"
    1318 "www.oceania.pool.ntp.org"
    1306 "www.north-america.pool.ntp.org"
    1285 "ntp.pool.ntp.org"
    1252 "asian.pool.ntp.org"
    1212 "north.pool.ntp.org"
    1163 "south.pool.ntp.org"
    1121 "e.g.pool.ntp.org"
    1014 "0.ol.pool.ntp.org"
    1000 "1.ol.pool.ntp.org"
     997 "3.ol.pool.ntp.org"
     980 "2.ol.pool.ntp.org"
     927 "0.vmware.pool.ntp.org1.vmware.pool.ntp.org"
     893 "www.1.centos.pool.ntp.org"
     891 "www.0.centos.pool.ntp.org"
     856 "www.0.asia.pool.ntp.org"
     639 "sg.cn.pool.ntp.org"
     548 "5.pool.ntp.org"
     445 "cn1.pool.ntp.org"
     411 "asis.pool.ntp.org"
     402 "china.pool.ntp.org"
     362 "-pcn.pool.ntp.org"
     320 "n.pool.ntp.org"
     320 "america.pool.ntp.org"
     215 "2.euleros.pool.ntp.org"
     205 "2.android2.pool.ntp.org"
     198 "0.euleros.pool.ntp.org"
     192 "2.android1.pool.ntp.org"
     190 "3.euleros.pool.ntp.org"
     185 "1.euleros.pool.ntp.org"
     178 "4.cn.pool.ntp.org"
     157 "172.130.192.250.cn.pool.ntp.org"
     136 "norch-america.pool.ntp.org"
     136 "1.librecmc.pool.ntp.org"
     129 "2.librecmc.pool.ntp.org"
     125 "qqqqqqq2.android.pool.ntp.org"
     121 "0.librecmc.pool.ntp.org"
     119 "aisa.pool.ntp.org"
     118 "3.librecmc.pool.ntp.org"
     103 "0.isoft.pool.ntp.org" 

NTP pool DNS polling efficiency

As mentioned earlier, the NTP pool uses DNS polling based on the pool subdomain to provide the required server IP to the client. DNSMon can measure the efficiency of its DNS polling by counting the DNS Record Set (RRset) frequency of A/AAAA records in the DNS.

In theory, if the load balancing is good, the chances of different IPs combining into RRset are equal. However, due to the influence of geographic location, service capabilities of different servers, and service strategies of different servers in actual operation, the number of different IP combinations for RRset vary greatly. We noticed that the RRset composed of different server IPs returned to the user by the NTP pool is very different.

Our data shows that there are 414,252 RRsets in 3,758 IPs, of which the top 4,000 RRsets (1%) accounts for 41.21% of the total number of records. The cumulative distribution graph of different RRsets is as follows:

What we learnt 

In summary:

1. When we have a massive amount of DNS data, various services that use DNS can be evaluated well. NTP pool is a typical example.
2. The number of server IPs in the NTP pool is around 4,000, and the number of users far exceeds 15 million.
3. In mainland China, servers participating in the NTP pool service are mainly concentrated in developed provinces and cities such as Guangdong and Beijing.
4. The access to the NTP pool subdomain is generally successful from a geographical point of view. However, there are large load differences between different servers in the region we studied.
5. About 3% of the domain names requested by the NTP pool are invalid, and almost all the invalid domain names have never provided time synchronization services.
6. DNS request data from the vendor-type subdomain of the NTP pool can be a good assessment of the size and business of specific vendors.

Adapted from original post which appeared on the 360netlabs Blog.

Zaifeng Zhang is a 360netlab security researcher.

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.