Reduce the effects of NAT for your IP blocklists

By on 22 Dec 2020

Category: Tech matters

Tags: , , ,

3 Comments

Blog home

IP blocklists are one of the simplest defences available to network operators.

However, little is known about their usage or potential unintended consequences when it comes to identifying malicious entities solely based on IP addresses, particularly those associated with address reuse techniques such as Network Address Translation (NAT) and dynamic addressing. Figure 1 illustrates the consequences of address reuse in blocklists.

Network illustration showing the consequences of address reuse in blocklists
Figure 1 — Users C and D share the same public IP address and user C is blocklisted. Due to address reuse, both users C and D are blocklisted. On the other hand, users A and B are in a dynamically allocated address space. User A, with IPA is blocklisted. Over time, user B can be allocated IPA, therefore it can potentially be blocklisted too.

One example of this happening in the real world is Cloudflare hosted websites blocking users because their NATed IP address was on a blocklist used by Cloudflare. To overcome this, Cloudflare’s best practices recommend users obtain a new IP address by either resetting their device or contacting their ISP. In reality, obtaining a new IP(v4) address may be impossible or too expensive for some users.

Network operators are largely in the dark about the effectiveness of blocklists…

In a recent paper presented at IMC 2020, my colleagues and I shared results of a survey that sought to understand if network operators are aware of such instances of excessive blocking due to address reuse and how they handle this issue.

Of the 65 network operators we surveyed:

  • 70% typically use their own curated blocklists. 
  • 85% also depend on external blocklists.
  • 59% use blocklists to directly block attack traffic. 
  • 35% use blocklists to divert traffic to more sophisticated systems, indicating that reused IP addresses can potentially cause more unintended damage.

We also asked network operators their perceived reasons for inaccuracies in blocklists with 56% saying NAT was at fault and 76% blaming dynamic addressing.

…but there are ways they can improve their performance 

In addition to the survey, we proposed two new techniques that network operators can use to identify reused addresses.

Using DHT crawler and RIPE Atlas probes to detect NATed and dynamic addresses

The first technique is to use a BitTorrent DHT crawler that uses BitTorrent messages to detect NATed addresses, which is illustrated in Figure 2.

Four-step illustration showing how BitTorrent crawler identifies reused addresses
Figure 2 — BitTorrent crawler to identify reused addresses.

This technique involves the crawler issuing get_nodes messages (1) to identify users with the same IP address in the BitTorrent network. However, not all users with the same IP address are NATed since the BitTorrent network can have stale information. As such, the crawler uses the bt_ping messages (2) to verify active users with the same IP address. The crawler sends out four bt_ping messages to four different ports with two different IP addresses, then gets responses (3) from two ports of the same IP address (IP 2) indicating NATing (4).

To detect dynamic addresses, we used the RIPE Atlas measurement logs. Every RIPE Atlas probe is allocated a unique ID and by using logs maintained at the RIPE central infrastructure, we were able to determine all IP addresses allocated to RIPE Atlas probes. We identified probes whose IP addresses changed frequently and thus determined IP prefixes that are dynamically allocated.

We then applied our detection techniques to identify reused addresses in 151 publicly available blocklists over 83 days. Our key observations are as follows:

Blocklists have reused addresses

There are 45K listings of NATed reused addresses and 30.6K listings of dynamically allocated reused addresses.

About 60% and 53% of all blocklists have at least one NATed and dynamically reused addresses.

On average, a blocklist lists 501 NATed IP addresses and 387 dynamic addresses.

Duration of listing reused addresses

Figure 3 shows the duration of all blocklisted addresses.

Graph showing duration of addresses in blocklists
Figure 3 — Duration of addresses in blocklists.

On average, blocklisted addresses are removed within nine days, and reused addresses are removed within ten days.

Among reused addresses, dynamically allocated reused addresses are removed much quicker. About 77% of all dynamically allocated reused addresses are removed within two days, compared to only 60% of NATed reused addresses and 42% of all blocklisted IP addresses.

Blocklisting NATed reused addresses can impact many users

Our crawler can identify several active BitTorrent users with the same IP address. Figure 4 shows the CDF of IP addresses and the number of active users. For most of the cases (68%), there are only two users. However, some IP addresses can have as many as 78 users that are in blocklists!

Graph showing the impact of blocklisted NATted reused addresses
Figure 4 — Impact of blocklisted NATed reused addresses.

Use a combination of greylists and blocklists

To reduce the impact of address reuse in blocklists, we also propose using a greylist containing a list of reused addresses that can be used for reducing the impact of excess blocking.

Network operators could use the greylist along with blocklists to divert traffic from addresses that are present in both the greylist and blocklist, or the blocklist maintainer could maintain a more accurate blocklist, by supplying a separate greylist that contains reused malicious addresses. In either case, it would be important to explore mechanisms for using blocklists as a defence mechanism when reused addresses are present in the IPv4 address space.

Our blacklist dataset (actively monitored) and the reused addresses that we detected during our study are publicly available.

Sivaram Ramanathan is a fifth-year PhD candidate at the University of Southern California.

Rate this article

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.

3 Comments

  1. paul vixie

    please stop calling them block-lists. that term was invented by a marketing department. the term you’re looking for is blackhole list. blocking was never the point.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Top