An often-overlooked source of data reflecting Internet performance and function is the data within its constituent networks. Often, this data is not collected or considered within a given network, let alone across and between networks. The TechArk Network Operator Measurement Activity (NOMA) aims to provide a neutral project platform to build a collaborative resource for Internet health metrics measurements. The first focus is on tracking the relative performance of IPv6 and IPv4 for HTTP traffic.
The plan
The approach of NOMA is to generalize and expand what some operators already do: take (simple) measurements from within operational networks to determine state and track trends of particular aspects of network function and experience.
These measurements are taken from multiple collection points spread across the access network. This gives a picture of the users’ experience of the network. (See this case study for a detailed description of an architecture for performance measurement in broadband networks). For a first effort, it made sense to look at the question of IPv4 versus IPv6 network performance for users, as this is a timely topic.
So, what are “simple” measurements? Using “libcurl” and http as the closest approximation of end-users’ experience, each collection point can gather:
- IPv6 DNS lookup to each target website
- Time to connect to the target website over IPv6
- Total time for each target website over IPv6
- IPv4 DNS lookup to each target website
- Time to connect to the target website over IPv4
- Total time for each target website over IPv4
With the timing numbers in hand, it’s easy to calculate a ratio of IPv4/IPv6 times as a simple test of whether IPv4 or IPv6 is performing better:
- < 1 means IPv4 activity is faster
- > 1 means IPv6 activity is faster
- =1 means IPv4 & IPv6 are the same
While the ratio is useful as a quick gauge of the state of affairs, it’s also useful to calculate the difference between the time for IPv4 and IPv6 tasks, as the ratio can blur important distinctions in total times.
Making it happen — RIPE Atlas
The measurements and calculations are simple — if you have the network from which to measure them. Absent that, and in order to illustrate the kind of results that are possible, NOMA has used the RIPE Atlas infrastructure to simulate the approach and to get real data. In our case, two US-based broadband networks were selected, and the RIPE Atlas probes within those networks were used to do measurements as if the networks were participating in the overall NOMA project.
There are some important differences between doing measurements using the RIPE Atlas infrastructure and running them from within one’s own network: all the http queries had to be directed to RIPE Atlas Anchors (that is, not generic websites such as ‘http://www.facebook.com’). Each probe reports its geographic coordinates.
Lacking any knowledge of network topology, this project used the probes’ coordinates to identify clusters of probes around geographic “pins” — each probe is clustered to its nearest pin, geographically. For the tests reported here, the pins were the geographic coordinates of the 12 RIPE Atlas Anchors that were used.
Some results
The measurements used for the data shown here were carried out every 30 minutes over a period of approximately one hour — that is, two tests per probe[1]. Clearly, longer-term measurements collection and review of data over time would be interesting, but they were beyond the scope of this phase of the project.
Each table shows the average responses from all responding probes in a cluster.
Each cluster towards a particular target
Here’s a sample of the kind of data one sees when having all probes measure towards a particular target. Pin 6 (the RIPE Atlas anchor identified in San Diego) was selected as the target.
IPv4/IPv6 Ratio
In the table below, green indicates that the ratio of the times (milliseconds) for DNS resolution, connection to the http server, to first byte, and overall execution is greater than one. That is, it took longer to complete the task over IPv4 than IPv6.
| Pin | Locality | DNS (v4/v6) | Connection (v4/v6) | Time to first byte | Execution (excl DNS) (v4/v6) |
|---|---|---|---|---|---|
| 0 | Atlanta | 0.934 | 1.061 | 1.055 | 1.0555 |
| 1 | Brookline | 7.155 | 0.958 | 0.994 | 0.994 |
| 2 | Chicago | 0.843 | 0.981 | 0.993 | 0.991 |
| 3 | Dallas | 1.331 | 1.116 | 1.125 | 1.125 |
| 4 | Denver | 0.896 | 0.928 | 0.980 | 0.979 |
| 5 | Los Angeles | 1.819 | 1.197 | 1.103 | 1.110 |
| 6 | San Diego | 0.737 | 1.249 | 1.453 | 1.471 |
| 7 | Coral Gables | 1.288 | 1.205 | 1.204 | 1.205 |
| 8 | Palo Alto | 12.826 | 0.372 | 0.646 | 0.648 |
| 9 | Phoenix | 0 | 0 | 0 | 0 |
| 10 | Ashburn | 4.459 | 0.812 | 0.833 | 0.832 |
| 11 | Seattle | 0.841 | 1.009 | 0.977 | 0.973 |
Difference
In the table below, green indicates that the difference of the times (milliseconds) for DNS resolution, connection to the http server, to first byte, and overall execution is greater than zero. That is, it took longer to complete the task over IPv4 than IPv6.
| Pin | Locality | DNS (v4-v6) | Connection (v4-v6) | Time to first byte (v4-v6) | Execution (excl DNS) (v4-v6) |
|---|---|---|---|---|---|
| 0 | Atlanta | -2.307 | 4.103 | 7.504 | 7.602 |
| 1 | Brookline | 139.178 | -4.178 | -1.251 | -1.163 |
| 2 | Chicago | -3.631 | -1.315 | -1.003 | -1.226 |
| 3 | Dallas | 13.031 | 5.860 | 12.867 | 12.894 |
| 4 | Denver | -2.560 | -3.732 | -2.291 | -2.370 |
| 5 | Los Angeles | 16.845 | 3.403 | 4.084 | 4.437 |
| 6 | San Diego | -21.130 | 4.227 | 14.656 | 15.328 |
| 7 | Coral Gables | 8.028 | 16.240 | 32.794 | 33.066 |
| 8 | Palo Alto | 216.735 | -48.686 | -37.315 | -37.289 |
| 9 | Phoenix | 0 | 0 | 0 | 0 |
| 10 | Ashburn | 75.019 | -19.237 | -34.819 | -35.239 |
| 11 | Seattle | -3.454 | 0.372 | -2.031 | -2.346 |
Each cluster towards its closest target
Here’s a sample of the kind of data one sees when having all probes measure towards only a target that is (geographically) closest to itself. This is an approximation of “near” in network terms, although it’s impossible to be certain of accuracy without any knowledge of topology.
IPv4/IPv6 Ratio
In the table below, green indicates that the ratio of the times (milliseconds) for DNS resolution, connection to the http server, to first byte, and overall execution is greater than one. That is, it took longer to complete the task over IPv4 than IPv6.
| Pin | Locality | DNS (v4/v6) | Connection (v4/v6) | Time to first byte (v4/v6) | Execution (excl DNS) (v4/v6) |
|---|---|---|---|---|---|
| 0 | Atlanta | 1.371 | 1.401 | 1.465 | 1.468 |
| 1 | Brookline | 2.521 | 1.025 | 0.990 | 0.999 |
| 2 | Chicago | 1.131 | 0.891 | 0.941 | 0.939 |
| 3 | Dallas | 0.806 | 2.300 | 1.397 | 1.396 |
| 4 | Denver | 4.088 | 1.292 | 1.562 | 1.554 |
| 5 | Los Angeles | 0.970 | 1.178 | 1.197 | 1.198 |
| 6 | San Diego | 0.737 | 1.250 | 1.453 | 1.471 |
| 7 | Coral Gables | 2.377 | 2.668 | 2.592 | 2.588 |
| 8 | Palo Alto | 17.318 | 0.154 | 0.346 | 0.347 |
| 9 | Phoenix | 0 | 0 | 0 | 0 |
| 10 | Ashburn | 1.760 | 1.037 | 1.056 | 1.031 |
| 11 | Seattle | 5.008 | 1.090 | 1.065 | 1.064 |
Difference
In the table below, green indicates that the difference of the times (milliseconds) for DNS resolution, connection to the http server, to first byte, and overall execution is greater than zero. That is, it took longer to complete the task over IPv4 than IPv6.
| Pin | Locality | DNS (v4-v6) | Connection (v4-v6) | Time to first byte (v4-v6) | Execution (excl DNS) (v4-v6) |
|---|---|---|---|---|---|
| 0 | Atlanta | 10.956 | 7.158 | 16.850 | 17.052 |
| 1 | Brookline | 32.148 | 0.602 | -0.521 | -0.072 |
| 2 | Chicago | 2.557 | -2.329 | -2.569 | -2.714 |
| 3 | Dallas | -11.495 | 63.726 | 39.447 | 39.554 |
| 4 | Denver | 86.375 | 4.061 | 17.540 | 17.606 |
| 5 | Los Angeles | -1.815 | 2.555 | 6.180 | 6.325 |
| 6 | San Diego | -21.130 | 4.227 | 14.656 | 15.328 |
| 7 | Coral Gables | 19.840 | 26.353 | 53.962 | 54.141 |
| 8 | Palo Alto | 330.7820 | -100.416 | -89.477 | -90.013 |
| 9 | Phoenix | 0 | 0 | 0 | 0 |
| 10 | Ashburn | 70.914 | 0.917 | 3.043 | 1.729 |
| 11 | Seattle | 80.526 | 1.300 | 2.125 | 2.143 |
These tables share enough data to be tantalizing. Network operators would naturally ask what caused the numbers above (traceroutes, which would be helpful for answering those questions, were collected but not analyzed). Since this is, effectively, a single snapshot in time, other questions arise as to whether the errors were transient and if the performance numbers are stable.
Discussion
These results are pretty preliminary, but they do open the door to further study. For instance, it seems the performance of IPv6 is better when measuring to a “near” target. One hypothesis is that performance improvement is because transit networks are not as friendly to IPv6 traffic as access networks. Other hypotheses are also possible, and only testing will tell.
Also, as it stands, the RIPE Atlas probe coverage is quite low (a few hundred probes in one network, and a couple dozen in the other). They are unevenly distributed throughout the networks, as the probe network is grown organically (as opposed to being planted by the operator for measurement purposes).
Nevertheless, while these results are pretty preliminary, they do highlight the value of the in-network perspective on IPv4 and IPv6 performance, and motivate further study. For instance, it seems the performance of IPv6 is better when measuring to a “near” target. One hypothesis is that performance improvement is because transit networks are not as friendly to IPv6 traffic as access networks. Other hypotheses are also possible, and only testing will tell.
For more information about the NOMA project, stay tuned to http://www.techark.org/noma .
[1] These particular measurements were run on July 16, 2017.
Leslie Daigle is Principal at ThinkingCat Enterprises.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.
Just wondering if the probes used were connected using native IPv6 and native IPv4 (dual stack) at least in the access network. Sometimes it explain the performance differences, while in this case, looks strange, because if using for example a TB typically the path which is better will be when the destination is close the the TB end-point or well connected to it.
If it is explained already in the post, I’m missing it.
Jordi,
The point of the measurements was to simulate the user experience. As such, there is no distinction made between probes that are on native v4 or v6 networks, or even whether they are behind some other kind of NAT. Yes, results would be a “cleaner” representation of absolute performance over IPv4 or IPv6 if the distinction was made, but that wasn’t the aim of the research.
Just like when the IPv4 Internet was really getting off the ground, there are undoubtedly some partly or mostly broken IPv6 configurations that skew such results. Give it time. IPv6 is ready for prime time. Enterprise networks and some big content owners (amazon, twitter, spotify, ebay, craigslist, paypal, …)
Indeed — an interesting question is how these probes’ numbers evolve over time (hopefully improving).