Let me tell you a story on how a subdomain of mine managed to serve 390,000+ PDF books without any of my servers being compromised.
When people talk about a site being compromised, usually you would think that your server has been compromised. That would be someone gaining access to your server and then doing whatever they please.
That didn’t happen here.
I take security very seriously. I have SSH locked down to only allow SSH key-based logins and even root logins are disabled. My site is static too, which means it’s only being hosted through nginx from a non-root user.
The only way someone is going to gain access to my server is if they manage to gain access to my workstation and steal my SSH key pair. The odds of that are remote because my workstation never leaves my office and I have the reflexes of a highly-trained ninja.
So how did a subdomain of mine end up help distributing 390,000+ PDF books without my server being compromised? Well, that’s easy…
How did it happen?
It boils down to this. About two years ago I was recording a video course that dealt with setting up HTTPS on a domain name.
In all of my courses, I make sure to ‘really’ do it on video so that you can see the entire process from end to end.
Back then I used nickjanetakis.com for all of my courses, so I didn’t have a dedicated domain name for the course I was working on, such as diveintodocker.com.
I also didn’t have a spare domain name that I wanted to publicly share, so I registered a new DigitalOcean droplet to host an example site on. Then I set up an A record to point ssl.nickjanetakis.com to that droplet’s IP address.
Cool, there’s nothing wrong with that — set up a temporary site for recording the course and then delete it afterwards. Easy peasy, and that’s exactly what I did but I forgot to remove the A record when I was done.
So for years, I had ssl.nickjanetakis.com pointing to an IP address that I was no longer in control of. That means the owner of that IP address could host anything and it would automatically be mapped to ssl.nickjanetakis.com without me knowing.
I ignored the first warning sign
I have Google Alerts set up so I get emailed when people link to my site. A few months ago I started to receive an absurd amount of notifications, but I ignored them. I chalked it up to ‘Google is probably on drugs’.
Stranger things have happened with Google, so I thought maybe something got mixed up. Now I’ve learned my lesson. While bugs roll out into production all the time, Google Alerts, being totally busted, is super unlikely.
I didn’t ignore the second warning
Part of my morning routine is to check emails with intent to answer any questions about my courses that may have happened during the night.
I usually skim the subject lines to see which emails to answer first but one of them caught my eye. It read “Hi, seems your website has been compromised”.
Well, you don’t see that every day. I figured it was spam that somehow made it to my inbox but I recognized the email because it was someone who signed up for one of my courses.
He sent me a screenshot showing a few PDFs being hosted from ssl.nickjanetakis.com, so I immediately went to Google and searched for site:ssl.nickjanetakis.com.
Figure 1 — Search results for the compromised subdomain.
It was mostly college books but there was a ton of other stuff too. Hopefully, by the time you read this most of them have been removed from Google’s index.
Fixing the problem in a few seconds
Since it was linked to a subdomain, I instantly knew what was wrong, especially because I remember using that subdomain a few years ago when I made that course.
The fix was really easy. I just hopped over to my domain registrar’s DNS settings and deleted the entry that mapped the IP address to that subdomain.
But before deleting it, I copied the IP address so I could open a support ticket on DigitalOcean. I figured they would like to know that someone is illegally distributing content on one of their servers. Now that they know the IP address, they can shut it down.
Avoiding the problem in the future
Always remove unused records from your DNS settings when you’re done using them.
DigitalOcean and many other cloud providers have purchased blocks of IP addresses and they provide these IP addresses to people like you and me.
When a droplet gets destroyed, the IP address eventually gets put into the public pool of available IP addresses and someone else will get it.
This is pretty scary because things like the above can happen if you’re not careful, but it also means if an IP address were blacklisted for doing something questionable, you might end up with that IP address in the future (but that’s a totally different problem).
Domain validation should be more strict
I think this brings up an interesting question. Right now you can validate you own a domain by putting an HTML snippet on your page. Services like Google Analytics allow for this.
Technically the person who took control over ssl.nickjanetakis.com could have proven ownership of that subdomain if they set up a page and hooked it up to Google Analytics.
Also, Let’s Encrypt’s web server based challenge would have passed.
I know I made a stupid mistake by not removing the A record but this could happen to anyone. I would like to see more services only allow for DNS-based authentication by adding TXT records.
Although I suppose the bigger problem here is having IP addresses being recycled. Hopefully, once IPv6 is fully in use we’ll have a big enough pool so that hosting providers can remove previously used addresses from their pool. That won’t be foolproof, but it’s a start.
Has this ever happened to you? Let me know in the comments below!
Original post appeared on nickjanetakis.com
Nick Janetakis has been programming for 20 years. He is a full stack developer, self taught and self employed as a freelance developer and teacher.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.
Thanks for the wonderful post information very useful guide.