Internet Exchange Points (IXPs) originally aimed to keep local traffic local and reduce dependence on third parties. However, ever-increasing traffic volumes create pressure for more dense and diverse peering, which challenges the traditional IXP model.
A fundamental shift in peering practices is remote peering (RP): networks may establish remote or indirect peering connections at IXPs, either over a ‘long cable’ (owned or leased) or over resellers that provide both IXP ports and layer 2 access. RP has fired up a heated debate within the operators’ community about its effects on Internet economics, resilience and performance.
Currently, we lack the tools to identify remote members and the implications of RP. To bridge this gap, we — FORTH in collaboration with the University of Crete, DE-CIX, TU Berlin, AMS-IX and Lancaster University — developed and tested a methodology to accurately identify RP, aiming to enable a more transparent peering ecosystem.
Key points:
- Detecting remote and indirect peers is challenging; for example, measuring their latency is not enough.
- We design a context-specific methodology, and infer RP with high accuracy (>95%) and coverage (93%).
- RP is a significantly common interconnection practice for ISPs; for the largest IXPs, remote peers account for 40% of their member base.
- Today, RP contributes twice as much as local peering to IXP growth.
Challenges in identifying remote and indirect peers
IXPs don’t always know or publish which of their members are remote; even the available data (for example, in PeeringDB) can be unreliable.
The main method of identifying remote peering is to measure the round-trip time (RTT) between a vantage point inside the IXP and the IXP peering interface of a member. If the RTT is above a certain threshold, typically 10 ms, then the member is considered to be remote. However, in today’s IXP landscape latency alone does not suffice:
- Using a single latency threshold heavily misinfers remote peering. Our measurement study shows that a threshold of 10 ms infers 40% of the remote members as local. Reducing the threshold to 1 ms still misinfers 18% of remote members, while reducing it further would misinfer local members.
- IXPs are not monolithic infrastructures. Instead, many IXPs are distributed among multiple co-location facilities across multiple cities or even economies. A member Autonomous System (AS) may be physically present in one of the IXP’s facilities and still have high latency to another IXP location. The Delay Matrix of NETIX, a wide-area IXP distributed across 22 European cities, illustrates that the pairwise latency between its facilities ranges from 1 to 60 ms.
Peering aspects
To address the aforementioned challenges, we introduce a ‘first-principles’ approach, which combines RTT measurements with technological and economic aspects that define the peering paradigm.
Port capacity: Typically, physical port capacities range between 1GE and 100GE. Connecting through a reseller allows IXP members to purchase virtual ports at fractional capacities at lower prices compared to physical ports. Since port capacities lower than 1GE can be obtained only through resellers, we use it as an indication of indirect peering.
Presence at co-location facilities: To establish a direct connection (local peering) to an IXP, an AS needs to deploy routing equipment in at least one co-location facility where the IXP has deployed switching equipment. We collect the facility list (including geographical coordinates) from PeeringDB and Inflect, and the websites of the 50 IXPs with most AS members. IXP websites provide additional facility data for 48% of the IXPs, allowing us to compile a more comprehensive data set.
Assuming perfect knowledge of port capacities and co-location data, identifying RP would be a straightforward lookup process. However, the available data are incomplete and noisy. To resolve such cases, we investigate two additional routing features.
Multi-IXP routers: An AS may connect to multiple IXPs through the same border router to reduce operational costs; we call such routers multi-IXP routers. We distinguish three cases: (i) When multiple IXPs are present in the same facility, a co-located AS may connect directly to all of them using a single router. (ii) Remote peers may connect through the same reseller to multiple remote IXPs where this reseller is present. (iii) An AS may connect with the same router to both local and remote IXPs, if, for example, it is co-located with one IXP and uses a reseller for another.
Private connectivity: Two networks co-located at the same IXP-hosting facility can interconnect directly with each other (private peering) without using the IXP infrastructure, for example, to reduce costs in case they exchange large volumes of traffic. When an IXP member appears to be privately connected with several ASes co-located at the same facility, this is a strong indication that this member is local to that facility.
To identify multi-IXP routers and private peering interconnections we analyzed traceroute measurements collected through the RIPE Atlas platform. Multi-IXP router interfaces appear in different traceroute paths to be interconnected with different interfaces of different IXPs, while private peers are identified by searching for interconnections between interfaces of different ASes without any intermediate IXP interface. To infer IXP interconnections in traceroute paths we use the traIXroute tool.
Inference algorithm
Step 1: Find reseller customers via port capacities. For each IXP member, we obtained the capacity of its peering port(s), through the IXP website or PeeringDB. If the member’s port capacity was lower than the minimum capacity of the physical ports reported in the pricing section of the IXP website, we inferred that it was connected to the IXP through a reseller and we identified it as an indirect peer.
Step 2: RTT measurements. We compiled a list of publicly accessible looking glass servers (LGs) and RIPE Atlas probes which reside inside IXPs and whose exact location is known. From these vantage points (VPs) we repeatedly pinged the IXP IP interfaces of each IXP member every two hours for two days. To reduce sensitivity to network conditions we store the minimum RTT value for each interface.
Step 3: Co-location-informed RTT interpretation. For each IXP interface, we calculated a geographical area around the VP location where the IP interface of the IXP member can be located, based on the minimum RTT. To calculate this area, we estimated the maximum and minimum distance between the VP and the interface based on the RTT. The presence of a facility of the IXP in this area denotes a local member.
Figure 1 shows a measurement from a VP in NL-IX in Amsterdam to an IXP member. The green-shaded ring denotes the feasible geographical area where the member can be located, according to a minimum RTT measurement of 5 ms (d1 is the largest possible distance and d2 the shortest). By checking the facility data both of the IXP and its member we determined that it is located either in Frankfurt or London. Therefore, even for distributed IXPs we can infer members that are directly connected to the IXP despite high RTT measurements.
Figure 1 — Combining RTT measurements with co-location data to infer the possible location of IXP members.
Step 4: Multi-IXP router inference. The previous steps may not be able to infer the peering type due to missing facility data or RTT measurements. In such cases, we used the multi-IXP router feature. We parsed the traceroute paths, and for all the IP interfaces of the same AS that are connected to IXP interfaces we performed alias resolution to group them to routers. If we found a router connected to multiple IXPs, and we had inferred the AS as local or remote to one of these IXPs from a previous step, we extended the inference to the rest of the involved IXPs based on whether they share co-location facilities or not.
Step 5: Localization of private connectivity. If an AS had private peers over the same router through which it is connected to an IXP, and the private peers are physically collocated to IXP facilities, we inferred that the AS is also local to the IXP.
90% of IXPs serve remote peers
We executed a ping measurement campaign from 7 to 9 April 2018 from each LG and RIPE Atlas VP to the peering interfaces of the IXPs that host the VPs. For the same period, we parsed the publicly available RIPE Atlas traceroutes, and applied our methodology.
Figure 2 shows the final inference results for the 30 largest IXPs in terms of the number of member ASes. Overall, we found 28% of the IXP interfaces for which we made an inference to be remote, with 90% of the IXPs having more than 10% of their peers as remote. Moreover, almost 40% of the members of the two largest IXPs (DE-CIX and AMS-IX) are remote.
Figure 2 — Inference results per IXP.
As shown in Figure 3, 75% of IXP interfaces are within 2ms from their respective VP, while more than 20% of the interfaces have minimum RTT over 10ms, a 2-fold increase since 2014. Interestingly, 6% of the peers inferred as remote/indirect claimed local presence in at least one IXP facility.
Figure 3 — Minimum RTT for each responsive IXP peering interface.
Based on RTT results and port capacities, almost 60% of these peers are indeed local in an IXP facility but prefer to connect through a reseller, while for the remaining 40%, the claimed facility presence is likely spurious.
Methodology yields ~95% accuracy
To validate our results, we obtained data from IXP operators through direct communication and IXP websites. Note that IXP operators usually only know whether their members are connected through resellers, and not where they are located, or if they use a layer 2 carrier to access their co-location facilities (a primary motivation for our study).
As shown in Figure 4, our approach yields ∼95% accuracy and precision, and the results are consistent across IXPs.
Figure 4 — Validation results per IXP in our test validation data set.
Preliminary insights and next steps
To understand the evolution of RP over time, we collected longitudinal data between 4 July 2017 and 10 September 2018 for 5 IXPs (LINX, HKIX, LONAP, THINX, and UAIX) and calculated the aggregate growth and departure rates per peering type.
In Figure 5, we observed that the number of remote peers grows twice as fast as the number of local peers. However, remote peers also exhibited higher (+25%) departure rates than local ones. These results indicate that IXPs that service most of their country-level peering ecosystems mainly expand their market pool by attracting remote peers, but such peers do not commit substantial resources, making it easier for them to terminate their IXP connectivity.
Figure 5: The increase in remote peers is 2x that of local peers.
In terms of RP impact on resilience, we found that 20% of the remote peers rely on multi-IXP routers, meaning that the AS and IXP-level peering diversity levels of such peers are misleading indicators of their resilience, since all of their interconnections depend on the same physical equipment.
As future work, we plan to:
- Investigate such resilience implications in depth.
- Examine traffic ratios of remote versus local peering interconnections at IXPs.
- Scale up our inference methodology by extracting RTTs from traceroutes (crossing IXPs) instead of pings (from within the IXPs).
For more information on the work please read the full paper published at IMC 2018 and watch our presentation at RIPE 77, or leave us a comment below.
A demo of our methodology is available at: http://remote-ixp-peering.net
If you like to help this work please contact us and consider contributing validation data and/or feedback.
Acknowledgements: This work has been funded by the European Research Council grant agreement no. 338402.
Contributors: George Nomikos, Vasileios Kotronis, Pavlos Sempezis, Petros Gigis, Lefteris Manassakis, Christoph Dietzel, Stavros Konstantaras, Xenofontas Dimitropoulos
Vasileios Giotsas is a researcher at Lancaster University. His work focuses on systems security and resilience, distributed systems, and measurement and analysis of Internet protocols.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.