IP blocklists are one of the simplest defences available to network operators.
However, little is known about their usage or potential unintended consequences when it comes to identifying malicious entities solely based on IP addresses, particularly those associated with address reuse techniques such as Network Address Translation (NAT) and dynamic addressing. Figure 1 illustrates the consequences of address reuse in blocklists.
One example of this happening in the real world is Cloudflare hosted websites blocking users because their NATed IP address was on a blocklist used by Cloudflare. To overcome this, Cloudflare’s best practices recommend users obtain a new IP address by either resetting their device or contacting their ISP. In reality, obtaining a new IP(v4) address may be impossible or too expensive for some users.
Network operators are largely in the dark about the effectiveness of blocklists…
In a recent paper presented at IMC 2020, my colleagues and I shared results of a survey that sought to understand if network operators are aware of such instances of excessive blocking due to address reuse and how they handle this issue.
Of the 65 network operators we surveyed:
- 70% typically use their own curated blocklists.
- 85% also depend on external blocklists.
- 59% use blocklists to directly block attack traffic.
- 35% use blocklists to divert traffic to more sophisticated systems, indicating that reused IP addresses can potentially cause more unintended damage.
We also asked network operators their perceived reasons for inaccuracies in blocklists with 56% saying NAT was at fault and 76% blaming dynamic addressing.
…but there are ways they can improve their performance
In addition to the survey, we proposed two new techniques that network operators can use to identify reused addresses.
Using DHT crawler and RIPE Atlas probes to detect NATed and dynamic addresses
The first technique is to use a BitTorrent DHT crawler that uses BitTorrent messages to detect NATed addresses, which is illustrated in Figure 2.
This technique involves the crawler issuing get_nodes messages (1) to identify users with the same IP address in the BitTorrent network. However, not all users with the same IP address are NATed since the BitTorrent network can have stale information. As such, the crawler uses the bt_ping messages (2) to verify active users with the same IP address. The crawler sends out four bt_ping messages to four different ports with two different IP addresses, then gets responses (3) from two ports of the same IP address (IP 2) indicating NATing (4).
To detect dynamic addresses, we used the RIPE Atlas measurement logs. Every RIPE Atlas probe is allocated a unique ID and by using logs maintained at the RIPE central infrastructure, we were able to determine all IP addresses allocated to RIPE Atlas probes. We identified probes whose IP addresses changed frequently and thus determined IP prefixes that are dynamically allocated.
We then applied our detection techniques to identify reused addresses in 151 publicly available blocklists over 83 days. Our key observations are as follows:
Blocklists have reused addresses
There are 45K listings of NATed reused addresses and 30.6K listings of dynamically allocated reused addresses.
About 60% and 53% of all blocklists have at least one NATed and dynamically reused addresses.
On average, a blocklist lists 501 NATed IP addresses and 387 dynamic addresses.
Duration of listing reused addresses
Figure 3 shows the duration of all blocklisted addresses.
On average, blocklisted addresses are removed within nine days, and reused addresses are removed within ten days.
Among reused addresses, dynamically allocated reused addresses are removed much quicker. About 77% of all dynamically allocated reused addresses are removed within two days, compared to only 60% of NATed reused addresses and 42% of all blocklisted IP addresses.
Blocklisting NATed reused addresses can impact many users
Our crawler can identify several active BitTorrent users with the same IP address. Figure 4 shows the CDF of IP addresses and the number of active users. For most of the cases (68%), there are only two users. However, some IP addresses can have as many as 78 users that are in blocklists!
Use a combination of greylists and blocklists
To reduce the impact of address reuse in blocklists, we also propose using a greylist containing a list of reused addresses that can be used for reducing the impact of excess blocking.
Network operators could use the greylist along with blocklists to divert traffic from addresses that are present in both the greylist and blocklist, or the blocklist maintainer could maintain a more accurate blocklist, by supplying a separate greylist that contains reused malicious addresses. In either case, it would be important to explore mechanisms for using blocklists as a defence mechanism when reused addresses are present in the IPv4 address space.
Our blacklist dataset (actively monitored) and the reused addresses that we detected during our study are publicly available.
Sivaram Ramanathan is a fifth-year PhD candidate at the University of Southern California.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.
Excellent well analysed and innovative article
Excellent
please stop calling them block-lists. that term was invented by a marketing department. the term you’re looking for is blackhole list. blocking was never the point.