Today, most Internet traffic is served by Akamai, Google, Meta, and Netflix. These hypergiants deliver most of the content through globally distributed servers. These servers can be on-net (within the hypergiant’s network) or off-net (hosted in other networks, typically ISPs).
Off-net servers benefit both hypergiants and ISPs. For ISPs, they reduce user latency and enable them to cut costs as less traffic now crosses network boundaries. Hypergiants generate revenue and reduce bandwidth demand on on-net servers. As such, studying off-net deployment strategies is crucial as they alter traditional Internet traffic flows and could be vital for next-generation technologies like 5G that require low latency.
While off-net deployments have been relatively well studied in the IPv4 Internet, their IPv6 footprint has remained unclear until recently. With IPv6 deployment gaining traction, a similar study on the IPv6 Internet was necessary. To address this, researchers from TUM, LAAS-CNRS, IPinfo, and I performed a first-of-its-kind study of IPv6 off-nets for 14 hypergiants, including major players such as Google, Meta, Netflix, and Akamai, and compared their deployment over IPv4.
We found IPv6 off-nets in 2,000 networks, compared to the more than 6,000 off-net networks for IPv4. Google, Facebook, and Netflix have the largest off-net footprints, deploying servers in as many as 1,300 IPv6 networks alone. Moreover, most IPv6 off-net deployments are in networks that have already deployed IPv4 off-nets.
As a whole, we found that instead of choosing to exclusively deploy IPv6 off-nets in ISPs where there are no IPv4 ones, hypergiants are trying to match their IPv4 deployments.
How do we uncover off-net deployments?
Researchers in 2021 put forward a novel methodology to uncover off-net deployments, revealing the IPv4 off-net footprint of several hypergiants. The idea was grounded in the observation that most of the current Internet traffic is encrypted and hypergiants being responsible for most of this traffic also actively rely upon TLS certificates to secure their data transfers.
By extracting relevant fingerprints from TLS certificates present on on-nets, researchers identified off-net servers of hypergiants. Additionally, to improve the reliability of the detection, other fingerprints such as HTTP(S) response headers specific to each hypergiant were also collected. Finally, comparing these two sets of fingerprints, extracted from TLS certificates and response headers, to those collected from an address outside the hypergiant allowed for identifying off-net servers.
While these fingerprints also exist for IPv6 traffic, there are several challenges with directly applying the methodology to IPv6. Firstly unlike IPv4, the address space is too large to enable collecting fingerprints from every single IP address. Secondly, as the whole address space can not be scanned, the collected on-net fingerprints have to be diverse enough to later detect as many off-net servers as possible.
We addressed these by performing our own measurements to state-of-the-art IPv6 hitlists consisting of responsive IPv6 addresses curated from a multitude of sources. We also enriched our hypergiant on-net fingerprints by merging those that we collected from popular domains, the whole IPv4 address space and the IPv6 hitlist addresses.
To validate our hitlist-based approach, we also performed other measurements such as those using the RIPE Atlas platform and the EDNS Client Subnet (ECS) DNS extension, which confirmed the best off-net coverage was achieved with the hitlist.
What fraction of networks per continent deploy off-nets?
We studied the per continent IPv6 off-net deployment of the top three hypergiants based on the off-net footprint (see Figures 1a to 1c).
Our findings show North America has the highest fraction of IPv6 off-net covered networks for Google and Netflix, with around 20% of all IPv6 networks serving users in the region. Meta has the highest IPv6 network coverage in South America, with more than 30% of the networks covered.
For all three top hypergiants, the lowest IPv6 coverage, ranging from 5% to 13%, is in Africa. IPv6 often outpaces IPv4, with Africa showing the most significance. As an exception to the IPv4 dominance, Meta has a higher IPv6 coverage in four continents, demonstrating Meta’s resolve to push IPv6 adoption.
Interestingly, we found that in South America, most networks deploy off-nets for these hypergiants, with nearly 700 networks using IPv6 and more than 1,500 using IPv4. Our findings reveal that this aggressive off-net deployment in South America stems from an underwhelming on-net presence of these hypergiants on the continent.
What fraction of Internet users have ‘access’ to off-nets?
Although the number of networks being covered by off-nets for a region might be small, the Internet user base served in the region can still be substantial if the off-nets are deployed optimally, that is, in networks that cater to a larger fraction of the user population.
For Google (Figure 2), while IPv6 user population coverage is exemplary in Oceania and Europe and above average in the Americas, it can be substantially improved in Asia and Africa.
Meta’s coverage is smaller than Google’s in most regions (see Figure 3). For instance, Russia is largely underserved by Meta’s IPv6 off-nets, which are present in only two Russian networks — Vimpelcom, hosting IPv4 and IPv6, and AS Rostelecom, hosting only IPv4 off-nets. These have a collective market share of less than 1% in IPv6 and IPv4. Deploying IPv6 off-nets in networks such as Baxet and MTS can improve IPv6 coverage in Russia by more than 50%.
In regions such as China and Iran, known for censoring Facebook and Google, we found off-nets in less than 7% of the networks. In contrast, economies such as the US and those in Scandinavia had off-nets in up to 41% of the networks. Interestingly, we also found 40% of Chinese networks with off-nets for the Chinese hypergiant Alibaba.
How does IPv6 off-net performance compare to IPv4?
We also evaluated and compared IPv4 and IPv6 off-net performance through latency-based measurements to reveal if IPv6 off-net deployments are at par with IPv4 ones, are catching up, or if significant room remains for improvement (see Figure 4).
Our findings reveal that the top hypergiants exhibited similar IPv6 and IPv4 performance for their off-nets, with most latency differences being under 5ms.
Challenges and future work
Although we uncovered IPv6 off-net deployments of several hypergiants, we acknowledge that our numbers for off-net hosting networks are a lower bound. This is due to the vast number of IPv6 addresses, which makes a total address space scan infeasible.
In the future, we intend to curate a more representative list of IPv6 addresses to test more IP addresses for off-nets, potentially uncovering more networks with off-nets.
Finally, we intend to continue tracking the growth of hypergiants’ IPv6 off-net footprint and perform a longitudinal study in the future.
To learn more about this work, read our ACM CoNEXT 2024 paper, A First Look At IPv6 Hypergiant Infrastructure.
Fahad Hilal is a Doctoral candidate at MPI-INF focusing on Internet measurements in the IPv6 Internet.
A version of this post was originally published at Internet Society’s Pulse Blog.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.