Bilguun Ganchuluun co-authored this post.
At Unitel LLC, we have developed an IP blacklist-checking tool, called AbuseBox, designed to monitor and verify IP addresses against blacklists, with the capability to submit delisting requests. Our research indicates that approximately 40% of all addresses find themselves on such blacklists, underscoring the importance of addressing this issue within our economy.
This post describes our basic goals for AbuseBox and how to use it.
Why are IP addresses blacklisted?
A blacklist is a method of creating a registry by detecting abnormal behaviour of users connected to the Internet on many sensors located around the world. Suppose suspicious behaviour is detected on the Internet. In that case, the sensors send a warning to the ISP or the address owner, and if the suspicious behaviour is repeated several times, the address is blacklisted.
Abnormal behaviours can include downloading or distributing copyright-violating or illegal content, intrusion attempts — sending spam mail or malicious traffic from a user’s device infected with a virus, attempting to attack other networks, and so on.
Risks of blacklists
However, blacklists are not without their own issues. The impact of being blacklisted can be significant for individuals, organizations, and businesses. Here are some potential consequences:
- Blocked email communication: If an IP address is blacklisted, emails sent from that address may be blocked or marked as spam by email providers, which can risk losing important mail.
- Difficulty accessing certain services: If an IP address is blacklisted, it may be blocked from accessing certain websites or services.
- Cybersecurity solutions use blacklist IP information to determine IP reputation. If your IP address is blacklisted, it means that the security solutions will block it. This means that it will affect direct mail and communication between organizations.
Furthermore, public addresses are frequently used as Network Address Translations (NATs), where the abnormal behaviour of one user can adversely impact numerous individuals. In our scenario, there are 100 users behind a single public address.
Research
During the research stage of AbuseBox’s development, we checked IP addresses blocked in the seven days prior to 28 June 2023 via AbuseIPDB. Mongolia ranked 54th in globally reported blacklisted IP addresses. During that week, 7,535 addresses were reported to AbuseIPDB, comprising 3.7% of the total. This figure was concerning, and we decided that action was needed.
We can determine the reasons for addresses reported to AbuseIPDB. Of the 25,244 addresses we checked, the top reasons are broken down in Figure 2.
For Mongolia specifically, we found nearly 40% of total IP addresses were blacklisted on 31 August 2023. As Figure 3 shows, we checked 156,976 addresses from the top 10 ASNs and 38.39% of these addresses were blacklisted. Figure 4 shows the sensors blacklisting Mongolian IP addresses.
Unitel’s solution
AbuseBox is our solution to regularly monitor IP addresses, return time-based reports, and automate delist requests to help ease the above risks from incorrect blacklisting. You can download AbuseBox from the GitHub repository.
Using AbuseBox, if one of your IP assets is found to be blacklisted, you can send a delist request immediately. Currently, however, it is only possible to send a delist request to a few sensors. In principle, it is technically feasible to send delist requests for most sensors and this is currently being worked on.
To check if an IP is blacklisted, a method known as ‘DNSBL’ is used, and information is currently available from 48 different sensors:
def check_dnsbl(reversed_ip: str, provider: str) -> Tuple[bool, str]:
query = f'{reversed_ip}.{provider}'
try:
response = socket.gethostbyname(query)
return True, response
except: # if name or service not know it means ip is not blacklisted
return False, ''Suppose the IP address 1.2.3.4 is listed in the blacklist database of a provider. To verify this, a socket connection is made to 4.3.2.1, which serves as the DNSBL provider. If a response is received, it means the IP address is indeed blacklisted. Conversely, if no response is received, it confirms that the IP address is not blacklisted.
View the full source code for DNSBL on GitHub.
Web application architecture
To make AbuseBox we used Django (REST framework) for the backend and PostgreSQL as the database (Figure 1). VITE + React js is used for the frontend stack (Figure 2).
Using AbuseBox
Using AbuseBox is simple. To obtain reports and use other features you must first log in, where you’ll be met with the ‘user panel’, shown in Figure 3.
Next, you can add a hostname (IP address or domain name) that you want to monitor and submit (Figure 8). This will add the host to the continuous monitor list (Figure 9).
A report example is shown in Figure 10. You can initiate a delist request directly from this panel.
Delisting
Manual delist can be solved in just one click. Firstly, we require a proxy to capture web requests. Then, we convert the captured request into code, utilizing the Python request function (Figure 7).
We’ve developed a delist API for a select few sensors; the source code is available on GitHub. If there’s no authentication process on the sensor side, you’ll need to convert the captured web traffic into code that sends a delist request:
import requests
url = "https://www.blocklist.de/en/delist.html"
headers = {
"Host": "www.blocklist.de",
"Cache-Control": "max-age=0",
"Sec-Ch-Ua": '"Not=A?Brand";v="99", "Chromium";v="118"',
"Sec-Ch-Ua-Mobile": "?0",
"Sec-Ch-Ua-Platform": '"Windows"',
"Upgrade-Insecure-Requests": "1",
"Origin": "https://www.blocklist.de",
"Content-Type": "application/x-www-form-urlencoded",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.5993.90 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
"Sec-Fetch-Site": "same-origin",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-User": "?1",
"Sec-Fetch-Dest": "document",
"Referer": "https://www.blocklist.de/en/delist.html?ip=",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.9",
}
payload = {
"ip": "1.1.1.1",
"action": "search",
"g-recaptcha-response": "",
"send": "delete",
}
response = requests.post(url, headers=headers, data=payload)
print(f"Response Status Code: {response.status_code}")Potential risks of delisting
While we’ve devised a method for automatically requesting delisting from blacklists, it’s crucial to identify and correct the underlying reason behind the IP address being blacklisted. Our approach involves sending web requests, but ultimately, it’s up to the sensor to determine whether to remove the IP address from the blacklist.
Before sending a delist request it’s important to remember that if you don’t fix the reason for the blacklist, you will be blacklisted again in a short time. Varying amounts of time may be needed for delist requests, so it’s best to avoid repeated requests.
You can download AbuseBox from GitHub. The delist API for select sensor source code is also available on GitHub.
Dashzeveg Baatartsogt is a Senior Information Security Analyst at Unitel LLC in Mongolia and actively volunteers with MNNoG and MNCERT.
Bilguun Ganchuluun is an Information Security Analyst (OSCP) at Unitel LLC in Mongolia.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.