Over the past year, I have been conducting a public survey to better understand how ISPs deploy IPv6. Thanks to everyone who participated in the study.
In this post, I will summarize some of the key findings thus far, as I presented at the RIPE 73 meeting last month (slides, video).
These findings include a comparison of how ISPs are deploying IPv6 to residential customers in different countries/RIR regions, as well as some observations about possible deployment mistakes or misunderstandings about IPv6.
Who responded?
As of 23 October 2016, 1,199 people had responded to the survey, of which 31% used IPv6 to participate in the survey (something I observed by looking at the logs from the apache server hosting the survey).
Figure 1. IP version of Survey Responder
Looking at whois and using other tools to correlate the data, I found that around 50% of those who responded are employees of ISPs, most of who responded from their own networks. Customers of ISPs responded typically from their residential network.
I have verified that almost every response is from an ASN that has both IPv4 and IPv6 allocations. This raises an interesting question: Why are 69% of ISP employees responding using IPv4?
This can be explained because in some situations, corporate LANs, even in ISP networks, have not yet deployed IPv6 in all their subnets. This is one of the first points we need to correct because the more users with technical knowledge and experience using IPv6, the faster we will be able to detect issues from broken IPv6 deployments.
Another interesting observation from the logs is that many users tried to connect to the survey with IPv6, but it reverted to IPv4. Checking the connectivity to those locations, it doesn’t look like IPv6 is much slower than IPv4, so it means that in some cases the Happy-Eyeballs mechanism is failing for other reasons, such as broken Path MTU Discovery.
This is something that I’ve observed on many occasions among data centres deploying websites with IPv6, either because they wrongly filter ICMPv6 or because some other mechanism, such as ECMP, silently breaks it. These problems point to a need for:
- More experienced IPv6 training to avoid creating the problem, and
- More complete IPv6 monitoring to discover these kinds of deployment mistakes.
It also begs the question if we should reconsider Happy-Eyeballs because of this problem of it hiding IPv6 deployment errors: we don’t have a similar mechanism in IPv4 and that’s why we discover, almost immediately, any network problem in IPv4!
Where did the responses come from?
The distribution of responses to the survey in different regions looks quite normal considering the degree of IPv6 deployment in each one.
Figure 2. RIR regions of survey respondents
There is a small exception to this distribution, with a high number of responses from LACNIC, in particular Brazil, from where we received 200 responses.
Figure 3. Respondents as per economy
If we analyse the responses comparing regions/countries, some countries in each region have a bigger IPv6 deployment, including the USA in the ARIN region, Belgium in the RIPE region, Brazil in the LACNIC region, and Japan in the APNIC region (there is almost no deployment in AFRINIC).
In terms of potential gaps in the data, we are missing responses from two countries with big Internet penetration in the APNIC region, South Korea and India.
This is one of the reasons why I will continue to run this survey to get a more detailed picture of IPv6 deployment in the APNIC region.
Take my survey on how you are deploying IPv6
What technologies are ISPs using to deploy IPv6?
Thirty-one per cent of respondents indicated their IPv6 service is already commercial and another 17% indicated it is still in trial stages.
Figure 4. Percentage of respondents who represent organization where IPv6 is commercially available
Figure 5. Preferred technologies used to deploy IPv6
Recent technologies such as FTTH (35%), xDSL (22%) and Cable/DOCSIS (20%) were among the most popular across the different regions. This result confirms that ISPs avoid investing in replacing CPEs, such as older DOCSIS 1 deployment, or first xDSL ones.
Looking at the evolution of the responses in previous months, and responses to questions on transition mechanisms, I observed an increase in the deployment of CGN. At the same time there is a small increase in the usage of transition technologies such as 464XLAT (and in a minor proportion MAP T/E), in order to provide IPv6-only access to residential customers, which is very consistent with what has been happening in cellular networks in the past few years.
Figure 6. Transition mechanisms respondents are using
Overall, responses related to the technical details of IPv4 or transition technologies are similar across the regions/countries surveyed.
Note: I don’t have results or any conclusions on whether there is any particular difficulty deploying IPv6 specifically related to access technologies or vendors.
WAN
With regards to the technical details of the WAN link, 61% are using a /64, and 13% are using unnumbered. The rest are using a /112, a /126, a /127, or other choices. However, looking closer at the details of the “other” responses, many of those who answered the survey appear to be non-technical people and may have interpreted the question incorrectly. Therefore, it could be likely that there are more /64 users.
When asked if their WAN prefix is stable (across reboots, and so forth, for the same customer in the same location), 18% responded “yes”, 11% responded “no”, and 71% responded “no answers”. My view on this question is that non-stability is related mainly to the provisioning systems, not something done on purpose.
With regards to what WAN addressing type they use, 63% of respondents indicated GUA, 24% link-local, 11% ULA and 2% other.
Figure 7. WAN prefix size
Figure 8. WAN prefix stable
Figure 9. WAN addressing type
Figure 10. WAN from same pool as customer prefix”
We also asked if the WAN is from the same pool as the customer prefixes, of which 7% responded “yes”, 9% “no”, and 84% “no answers”. Considering this large proportion of “no answers”, it is interesting that 66% of ISPs are using the first /64 from the customer’s prefix, while 34% are not.
This practice is a very convenient, non-standard way that is supported by almost every vendor. It was described in an Internet Draft in 2006, however, did not progress in the v6ops IETF working group so I let it expire. As a consequence of this survey, I’m intending on reviving it to seek the support of the RIPE IPv6 WG and make it a Best Current Practice.
Overall, there were no major differences in the answers to the WAN-related questions between regions and countries.
LAN
Below are the results from a similar set of questions as those above, this time regarding the LAN prefix.
With regards to what prefix is allocated for customers’ LANs, 22% of the respondents indicated that they are using a /48, 35% indicated they are using a /56, 33% a /64 and 10% other sizes (among them, a /60, a /62, a /57, a /127 and a /128). Some responses seem unreal (/29, /44) and (I’m guessing) were probably provided by non-technical responders.
Figure 12. LAN prefix size
The above results reflect a major allocation problem: the provision of a single /64 for customer LANs. This is a totally “broken” concept, as it implies that customers are not able to do subnetting in their networks, or arrange different VLANs, or different SSIDs (such as a guest one). The reason for this, most likely, stems from the concept that one address is enough because we have NAT, but this is only true in IPv4!
Again this is highlighting a lack of expertise, or even worst, that people doing IPv6 training are missing key points, which means that the people doing those deployments will need to spend time redoing it.
Looking at the distribution of those responses among different regions/countries, most of the “broken” deployments (/64) are being done in the LACNIC region and in a few APNIC countries. More “advanced” countries (in terms of IPv6 deployment) like those in northern Europe, Japan, Australia, New Zealand and China, are using a /48, while the rest of the countries in the RIPE and ARIN regions are using a /56.
Figure 13. LAN prefix stable
Figure 14. Customers can opt to have LAN prefix stable
Figure 15. Customers who pay extra cost for LAN prefix stability
With regards to customer LAN prefix stability (same prefix for the same customer in the same location, persistent across reboots, and so forth), 17% respondents answered “yes”, 11% “no”, and 72% “no answer”.
This is another example of having an “IPv4 mentality” when deploying IPv6 because we don’t see the need for an end user to have stable addresses. In IPv6, it is clear that if our customers have stable addresses, we are making it easier for our customers, ourselves and third parties, to develop, deploy and provide services, which in turn means increasing our network value and our turnover.
Again, correlating this data by regions, we can see that mainly AFRINIC, RIPE and APNIC are doing stable assignments to customers, while ARIN is mainly non-stable, and LACNIC is almost half and half.
We also asked if offering the stable prefix is an “extra” or “opt-in” option, and if this option means “extra cost”. As can be seen in the pie charts, the large majority provide an IPv4 service as well as an IPv6 one, and still there is an important proportion of ISPs providing a public IPv4 address.
Figure 16. IPv4 service provided
Figure 17. Public IPv4 address at CPE WAN
Figure 18. IPv4 address is stable
Figure 19. Customers can opt to have stable IPv4
Figure 20. Extra cost for stable IPv4
DNS
The last set of questions related to the DNS aspects, which have provided similar observations as those for prefix stability.
It is obvious that addressing devices or services inside the customer network can’t be easily done using IPv6 addresses, so having some kind of DNS “naming” service makes sense. For example, the customer can have IPv6 cameras names as “camera1.customer.ispname.com”, and so on.
Figure 21. Respondents’ organizations using IPv6 reverse DNS
Figure 22. NS delegation for stable IPv6 prefix
Figure 23. DNAME for non-stable IPv6 prefix for PTRs
Takeaways
The most worrying takeaway for me from this survey so far is that there are still a lot of engineers that try to deploy IPv6 the same way they deploy IPv4, which is plain wrong.
The main misunderstandings are related to the prefix size assigned to the customer and their stability.
One of the most common mistakes, in my experience, is that many ISPs get the default allocation from their RIR (/32 in most of them, /29 in the RIPE NCC), without preparing a proper addressing plan. This means that if the ISP is not a small one, or they are using some transition mechanism such as 6RD, a /32 can be too small to assign a “correct” prefix to the end-customer.
Furthermore, many ISPs aren’t doing the same degree of monitoring for IPv6 as they do for IPv4, so common errors are not being advertised.
With regards to common mistakes seen when deploying IPv6 on websites, there needs to be greater awareness about the effects of filtering of ICMPv6 on purpose, misconfiguring load-balancers or using technologies such as ECMP without taking measures to avoid breaking PMTUD. These may not directly be related to the deployment of IPv6 in networks but they do affect them because access technologies usually impact by reducing the MTU.
This is the first report from the survey and as mentioned I plan to keep it open and report on it in the coming years with a particular goal of targeting countries where we have collected very few or no responses. We also want to study the possible implementation changes when ISPs discover that they are doing something wrong or can improve the way they actually deploy IPv6.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.