IPv6 Performance – Revisited

This is an update of an article published in November 2015 – Examining IPv6 Performance. The update describes a new reporting tool that can show the performance of IPv6 relative to IPv4 at both a country-by-country level and a network-by-network level.

Every so often I hear the claim that some service or other has deliberately chosen not to support IPv6, not because of some technical issue, or some cost or business issue, but simply because the service operator is of the view that IPv6 offers an inferior level service as compared to IPv4, and by offering the service over IPv6 they would be exposing their clients to an inferior level of performance of the service.

But is this really the case? Is IPv6 an inferior cousin of IPv4 in terms of service performance?

Let’s try and answer that question by using measurement. In this article I’d like to report the results of a large-scale measurement of IPv4 and IPv6 performance looking at the relativities of IPv6 and IPv4 performance.

What is “Performance”?

In networking the term “performance” can have many aspects. Often we might characterize a network’s performance by its sustained carrying capacity, or bandwidth, or by its end-to-end delay, or its level of delay variation or jitter. Each of these parameters could affect an application’s performance. A large scale data transfer is affected by carrying capacity and by end-to-end delay, while a raw encoding of a voice or video stream could be more sensitive to jitter than to the end-to-end delay.

That’s a large collection of elements that can affect the network transaction. We can reduce this somewhat by refining the question. If we have two sessions running between the same endpoints, using the same end-to-end transport protocol and using the same applications at each end, performing the same transaction at much the same time, and we vary only the underlying IP protocol used by these two sessions, then to what extent do we see differences in performance?

There appear to be two readily measurable aspects of performance that can vary between the two protocols with the same endpoints that will impact on the end result.

The first is the reliability of the protocol in the face of active middleware. Are all connection attempts successful? Do we observe a higher connection failure rate in one protocol or the other? Is there middleware deployed on the network path that is dropping IP packets based on the IP protocol?

The second is path variation. A change in IP protocol may produce a change in the network path used by the packet.

 

This leads to the observation that for the purposes of comparing the performance of these two versions of IP, what we would like to look at is the relative levels of reliability of connection establishment and relative round trip times across a series of experiments where the other factors are held constant. To what extent does changing just the IP protocol from IPv4 to IPv6 have on the anticipated performance of the network transaction?

Measuring Performance

This measurement is based on data collected from end user measurements. The measurement technique uses an advertisement distribution network to load a measurement script on a user’s browser. The script fetches three invisible pixels, one using a URL that is only accessible using IPv4, the second with IPv6 and the third is accessible using either protocol. All of these pixels are loaded from the experiment’s servers, and a full packet log is recorded at each server. The actual measurements are performed through subsequent analysis of the server’s packet logs.

Connection failure is challenging to measure if the only available data is coming from the server, as the task has some aspects of attempting to measure something that does not in fact exist. However, we are using TCP in this measurement, and one aspect of connection failure is directly visible to the server. TCP uses a three-way handshake to start a session. Normally this handshake looks like Figure 2. The server receives an opening TCP packet with the TCP SYN flag set. The servers will respond with a TCP packet with the SYN and ACK flags set, and the connection is complete when the server receives an ACK packet.

 

One visible form of failure is a hung connection, where the server sees and responds to the opening SYN, but fails to receive the subsequent ACK to complete the connection. What we can measure for each protocol is the relative ratio of hung connections to successful connections. What percentage of all connection attempts fail to complete, and is this number different when considering IPv4 and IPv6 connections? This is not a complete picture of connection failures, as we may not even see the initial SYN of an attempted connection, but it is a useful comparative metric, in that it allows us to directly compare one aspect of connection robustness between IPv4 and IPv6.

The second measurable performance attribute is the end-to-end delay, measured as a round trip time. A log of a TCP session provides a constant record of round trip time measurements by looking in the server’s log at the time when data was sent and the time when that data was acknowledged. This implicit RTT estimate can be noisy, with factors such as delayed ACKs in the TCP protocol, and client side issues, such as internal job scheduling both in the host operating system and with the browser itself, and similar server-side issues. One way to avoid much of this added noise component to the underlying signal of the RTT measurement is to look at the time taken to complete the initial TCP connection in Figure 2, from reception to the initial SYN to the reception of the ensuing ACK. This TCP connection operation is normally performed within the operating system as a kernel function and is less prone to imposed jitter from the external factors, so this measurement offers us one of the clearer ways to view the end-to-end round trip delay.

Measurement Results

The major update from my previous report is the creation of an online tool that reports on performance on an ongoing basis. The tool is accessible at: http://stats.labs.apnic.net/v6perf.

The home page of this report is a world map, and a set of tables. There are two generic families of reports: a comparison of RTT measurements, comparing IPv4 and IPv6; and an absolute value of IPv6 connection robustness.

RTT Comparison

The “home” screen of the RTT comparison report is the world map, shown in Figure 3.

Each country is coloured based on a relative RTT metric. This metric is computed by looking at the two RTT times for each dual stack client (in milliseconds) and subtracting the IPv4 RTT from the IPv6 RTT. In the example shown at the start of this article, the IPv6 RTT is 213ms, while the IPv4 RTT is 315ms. The result of the subtraction is -102ms, indicating that in this particular case the IPv6 RTT is 102ms faster than IPv4. These individual measurements are collected for each country (the country is determined by geolocating the IPv4 part of the dual stack address pair), and the last 30 days’ measurements are used to generate an average value for each country.

The data used to colour the map is also reproduced in the tables in the second half of the page. This table is helpful in so far as often a single average value can conceal a wealth of information. What the table shows that the map cannot, is the number of sample points on which the map was based. For example, while the average RTT difference in the Dominican Republic was 93.4ms in favour of IPv6 being faster, this result is based on the measurement of a single, dual stack host. A more reliable number is drawn from Viet Nam, where averaged across 4,479 values, IPv6 was on average 36.07ms faster.

Control variables in the screen allow variation of the end date (the default is the most recently processed date, typically yesterday), and the averaging window (the default is 30 days). The threshold sample rate can also be raised to remove and report those countries whose total sample count in the window is less than the threshold.

Connection Robustness

The other form of world map is obtained by selecting the Plot Type of “Failure of v6 TCP connections”. Here the data reflects the relative ratio of failed IPv6 connections as compared to all IPv6 connection attempts. The map shows the average connection failure rate expressed as a percentage of all seen connection attempts over the specified time window for users who are geolocated to a particular country. An example screen shot of this view is shown in Figure 4.

There are many potential causes of IPv6 connection failures. A common reason is overly restrictive filters applied on the customer site, where incoming IPv6 packets are refused. It is also possible that there are asymmetric routing issues that allow the client to see the relevant experiment server, but not the reverse.

Other Data
As well as the data relating to the mean difference in measured RTT values, there are three additional columns in the tabular reports: “Dual Stack”, “Dual Stack (300ms)” and “v6 Use Rate”.

Let’s look at the Dual Stack figure first. As well as RTT times, we can also collect data on which protocol was used by the client to retrieve the dual stack object from the server. We can match this with the different RTT times recorded against this host to ask the question: “Did the client end up picking the fastest protocol to retrieve the dual stack object?”. In the tabular report, the percentage value is a variant of this question, namely: “What percentage of the time for all clients did the client use the fastest protocol to retrieve the dual stack object?”. Where the sample size is relatively high (more than 1,000 individual samples over a 30-day window), the ‘accuracy’ of the dual stack selection outcome is shown as a percentage of the total fetch. This is the “Dual Stack” column.

However, this Dual Stack selection figure probably misrepresents the true objective of the protocol selection algorithm in dual stack hosts. The aim of the “Happy Eyeballs” protocol selection algorithm is not to select the fastest protocol in all cases. The algorithms will prefer IPv6, and it does so by selecting IPv6 as long as it is not too slow as compared to IPv4. The RFC that describes this procedure, RFC6555, does not specify how slow is “too slow”, but it does note that Firefox and Chrome implementations of this procedure use a 300ms timer. The second column reports of the Dual Stack selection process, using a modified selection question: “What percentage of the time did the client use the fastest protocol, allowing IPv6 to have a 300ms handicap advantage, to retrieve the dual stack object?”.

There are some changes evident between these two questions. For dual stack users in Greece, for example (136,299 samples over 30 days), IPv6 is on average 3.85ms faster than IPv4. Users picked the fastest protocol 63% of the time. However, when IPv6 is allowed a 300ms “Happy Eyeballs” advantage, the result is 98% accuracy in selecting the fastest protocol.

The final column in the tabular reports in the v6 use rate as a percentage, showing to what extent there is IPv6 use as a percentage of all samples for IPv6.

Per-Country and Per-AS Reports

There are also reports for each country and each access network service provider, providing a similar report at the level of individual countries and individual networks.

The per-country reports also include a table of the network service providers who provide services in that country. The table provides the same details of the average RTT difference between IPv4 and IPv6, the IPv6 TCP connection failure rate, and the Dual Stack selection outcomes for each access network in the country that has recorded IPv6 activity.

It is possible to add a number of plots to the country and network reports. Three plots are added when the radio button for “Add Plot of Time series RTT Profiles to RTT Report” is ticked.

The first plot is the actual distribution of the individual RTT difference measurements across the same time window used in the table. This shows more information that is shown in a single average figure, displaying both the distribution of measurements and the cumulative distribution of measurements, as well as displaying the average and the mean standard deviation values.

The distributions displayed in these plots show that it is rare to see a smooth distribution. More common is a distribution such as that shown in Figure 5 (data for AS1221, Telstra), where there are a number of local peaks of difference in RTT values.

This distribution shows a level of variation in RTT measurements that is not visible in the average value, nor in the mean standard deviation. In the case shown in Figure 5, there is a local peak of some 1,000 sample points where the IPv4 path is some 40ms faster than the equivalent IPv6 path. In this case slightly more than 50% of the samples show a faster path using IPv6, with some clustering at 10ms and 24ms.

AS7018 in the United States (AT&T Services) shows an interesting distribution (Figure 6). Slightly over one half of the samples show IPv6 being no slower than 10ms than the accompanying IPv4 RTT value. However, there are three local clusters where IPv6 is respectively 15ms, 25ms and 75ms slower.

The second plot is a time series plot of daily values, showing the average and the mean standard deviation of the daily data set for each day. An example of this plot is shown in Figure 7.

The third plot shows the number of measurement sample points that have been gathered per day (Figure 8). The blue data points represent the number of IPv6 addresses that have completed a connection and are able to be associated with an IPv4 address in a dual stack scenario. The red data points are the total of all unique IPv6 addresses seen per day, including those addresses that fail to complete a connection. Teredo and 6to4 IPv6 addresses are not included in this count.

Conclusion

What can we conclude about the relative performance of IPv6 to IPv4 from these measurements?

These measurements show that in a large set of individual comparisons where the IPv4 and IPv6 paths between the same two dual stack endpoints are examined, the two protocols, as measured by the TCP SYN round trip time, are roughly equivalent on average, but with some significant outliers.

At a finer level of detail, there are discernible differences between geographic regions. IPv6 appears to be faster on average in Europe, Africa and Oceania, while IPv4 is faster in Asia, and the Americas. At a further level of detail these differences between IPv4 and IPv6 extend to individual access networks where some networks appear to handle IPv6 packets in an entirely different manner to IPv4 packets.

While the TCP connection performance is roughly equivalent once the connection is established, the probability of establishing the connection is not the same. The current connection failure rate for IPv4 connections was seen to be some 0.2% of all connection attempts, while the equivalent connection failure rate for unicast IPv6 is eight times higher, at 1.6% of all connection attempts.

So, with respect to the performance of IPv6, there is still some scope for improvement here.

These reports are available from APNIC Labs.

Rate this article

---

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.