At the recent SANOG meeting held in my homeland, Pakistan, I wanted to provide the local community with some insight into the importance of Internet Exchanges (IXs), specifically the need to host content locally.
Knowing that data is king among network operators, I set up a virtual machine as soon as I arrived to collect information on several key metrics, including latency and the hosting location of .pk domains. Needless to say, the results were surprising.
How long does it take to connect to public DNS services?
First, I tested for latency, specifically the time it takes to PING three of the most popular public DNS services: Cloudflare DNS (1.1.1.1), Google Public DNS (8.8.8.8), and Quad9 (9.9.9.9). (PING is not the best way to test DNS but this is for reachability purpose only.)
Before leaving my home in Sydney, Australia, I did the same to offer a comparison. As you can see from the results below (Figure 1), all were below 1ms.
The results for Pakistan were less consistent.
Cloudflare was the best of the bunch with an average of 3ms, thanks largely to the three data centres they opened in Karachi, Lahore and Islamabad late-2018.
traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 52 byte packets 1 110.18.48.1 (10.18.48.1) 5.901 ms 2.933 ms 3.664 ms 2 2202.163.100.105 (202.163.100.105) 5.297 ms 5.205 ms 7.234 ms 3 * * * 4 202.163.100.66 (202.163.100.66) 6.572 ms 4.340 ms 4.051 ms 5 119.159.240.53 (119.159.240.53) 3.728 ms 3.403 ms 3.399 ms 6 khi77.pie.net.pk (221.120.251.205) 9.160 ms khi275.p01.pie.net.pk (221.120.251.201) 5.765 ms 7.170 ms 7 rwp44.pie.net.pk (221.120.251.101) 5.268 ms 4.654 ms rwp44.pie.net.pk (221.120.251.88) 4.956 ms 8 static.khi77.pie.net.pk (221.120.200.3) 5.270 ms 5.438 ms 5.009 ms 9 one.one.one.one(1.1.1.1) 3.722 ms 3.915 ms 3.467 ms
Google was second and even though it has several caches hosted in Pakistan, its latency hovered around 25ms.
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets 1 10.18.48.1 (10.18.48.1) 7.683 ms3.913 ms 4.363 ms 2 202.163.100.105 (202.163.100.105) 4.654 ms 5.716 ms 6.581 ms 3 * * * 4 202.163.100.66 (202.163.100.66) 7.731 ms 5.862 ms 4.388 ms 5 119.159.240.53 (119.159.240.53) 5.593 ms 7.199 ms 7.226 ms 6 10.253.4.26 (10.253.4.26) 8.717 ms 10.253.4.8 (10.253.4.8) 6.228 ms 10.253.4.26 (10.253.4.26) 5.140 ms 7 74.125.118.170 (74.125.118.170) 25.506 ms 23.927 ms 22.310 ms 8 * * * 9 dns.google(8.8.8.8) 25.356 ms 24.182 ms 24.443 ms
Quad9, which has no local POPs, came in at a very slow 203ms.
traceroute to 9.9.9.9 (9.9.9.9), 64 hops max, 52 byte packets 1 10.18.48.1 (10.18.48.1) 2.921 ms 5.942 ms 3.498 ms 2 202.163.100.105 (202.163.100.105) 5.388 ms 5.041 ms 4.442 ms 3 202.163.100.25 (202.163.100.25) 5.492 ms 202.163.100.121 (202.163.100.121) 5.179 ms 5.183 ms 4 202.163.100.66 (202.163.100.66) 3.842 ms 4.455 ms 2.862 ms 5 119.159.240.53 (119.159.240.53) 4.525 ms 4.177 ms 4.179 ms 6 10.253.4.133 (10.253.4.133) 6.869 ms 4.925 ms 6.503 ms 7 et7-1-5.catania8.cat.seabone.net (93.186.129.38) 104.627 ms 104.593 ms 104.202 ms 8 ae24.franco31.fra.seabone.net (195.22.211.50) 199.860 ms 134.881 ms 137.868 ms 9 de-cix.woodynet.net (80.81.194.42) 249.100 ms 205.069 ms 204.172 ms 10 dns9.quad9.net (9.9.9.9) 203.927 ms !Z 201.411 ms !Z 204.876 ms!Z
What this experiment shows is the importance of hosting services locally. We could expect to see a further reduction in latency if such services were also hosted at a local IX, as it would reduce the number of hops for those that peer with it to one; the benefits of which would be reduced transit costs, with the savings being handed down to customers.
Currently, Pakistan has one IX, PKIX, which has two exchange points — one in Islamabad and one in Karachi — with 17 Members peering with it.
Where are ‘local’ websites hosted?
As we’ve seen so far, the best way to reduce latency for DNS services is make them local, so packets don’t have to travel extreme distances. The same can be said for web content too, whereby websites hosted locally should, in theory, be faster to connect to than those hosted internationally.
So, for my second experiment, I looked at where websites with .pk country code top-level domains (ccTLDs) were being hosted.
The time of my test was short so I could only get public data, which was around 15,000 .pk domains. However, like the latency test, it provided a very clear picture of the current situation.
Of the 14,927 .pk domains, only 1,541 are being hosted by ISPs or entities registered in Pakistan, and another 1,300 are being hosted by Cloudflare. That’s less than 20% of sites that, you would think, would be targeting local users. Instead, more than 80% of ‘local content’ is being hosted in servers outside of Pakistan, and, as such, is subject to international transit costs and increased latency.
Digging deeper, I also looked it up for gov.pk, who you would think are targeting the people in Pakistan, right? Out of 431 gov.pk domains I found in the 15,000 domain addresses, only 72 were registered or advertised by Pakistani entities, 28 are with Cloudflare, and the remaining 331 are outside of Pakistan.
Needless to say, I was quite surprised that so many local council websites are hosted outside of Pakistan, most likely because they were cheap.
Finally, I looked at edu.pk. For a university to host something outside of Pakistan and not within the university is blasphemy for me. Unfortunately, this is the case for the majority of edu.pk sites with fewer than 20% hosted locally.
How many websites have Quad A records?
The IPv6 Task Force was established in Pakistan around 2006/07 during which time it built a test network between several ISPs. Unfortunately, IPv6 never caught the limelight beyond testing and enabling in core networks, and capability remains less than 1%.
I was hoping that because many websites are hosted outside Pakistan, that content might have been accessible via IPv6. Unfortunately, this wasn’t the case either, with the exception of Cloudflare.
One thing that’s not clear from Figure 5 is that of the 1,304 websites hosted by Cloudflare, why were only 1,087 responding to AAAA requests? Have some turned off IPv6? If so, why? I plan on digging deeper into this using the virtual machine generously provided by Rapid Compute (a cloud services provider in Pakistan).
Finally, it’s worth pointing out that all of the above would be a lot worse if it weren’t for Cloudflare setting up its Content Delivery Networks (CDNs) in Pakistan in 2018. However, if more CDNs are set up in Pakistan, there needs to be more local content available for them to host. This surely cannot be too difficult for an economy with more than 200 million people.
Aftab Siddiqui is a Technical Engagement Manager for the Internet Society.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.
Very apt and interesting piece emphasising the
Critical role of hosting of content locally near the Exchange points for enhanced consumer experience .
Key Illustrations are beautifully depicting the need for
more and more CDNs and DNS to be with in the country.
A partial viewpoint with confusing analysis.
Establishing CDN/PDN Data Centers in Pakistan is the only solution
Hmm
Impressive analysis by typical Aftab 😉 but the issue is that first what are the opportunities for international businesses by hosting their contents locally in Pakistan? Second, Pakistan is not very popular as a whole in deploying security counter measures in almost every organization and thus is a source of large amount of malicious traffic and noisy IPs.
Thus, I personally experienced many well known EU based websites that blocks connection request originated from Pakistan. So, the dynamics regarding Pakistan are way different than rest of the world.
Thanks, Mr Aftab for sharing this analysis.
It is eye-opening for gov and privacy concern orgs.
In today’s world, every country wishes to host their data inside their physical boundaries. There are various motives behind this act, kill privacy concerns and make data available for locals in case of DDoS attack, are prominent justifications.
Pakistan become famous if people start hosting all data in Pakistan and it automatically opens all banned IP as well. It also kill privacy concerns.
Security countermeasures implemented by hosting provider is not good so for but it can be improved with time.
Data hosted in Pakistan in any case beneficial in terms of new jobs, easy reachability.
A very good article. I had never thought about this part of networking. But there are few problems with hosting data centers in pakistan. One power supply. the power supply must be consistent. Second weather, the weather of the northern areas is good but we dont have good power supply or connectivity there. where we do have connectivity we dont have the right conditions that is cheap electricity and stable electricity.
I have a general question!
I am facing latency issues while accessing VMWare Horizon VDI from Pakistan. And at times it completely freezes on me, and when I ping I get request time out after every 4 or 5 successful pings. Is there anyway I could overcome this?
Thank You