Border Gateway Protocol (BGP) best paths on which the Internet relies are not always the ‘best’ when it comes to bandwidth and latency. This is due to several factors including BGP not considering congestion on links and AS-PATHs using prepending no longer reflect closeness.
As such, engineering egress traffic toward neighboring Autonomous Systems (ASes) is a constant challenge for network operators seeking better paths.
To begin with, the potential benefit of egress traffic engineering is unclear. Measuring the performance of a non-best path to a destination requires preferring the non-best path. However, using local preference to do this impacts all user traffic to the destination.
In an effort to improve our understanding of this, we experimented with Segment Routing (SR) based BGP Egress Peer Engineering (BGP-EPE). We measured non-best paths without impacting user traffic by relying on encapsulation to a chosen exit. We showed that 77% of prefixes have alternate paths of shorter latency than best paths and about 60% of prefixes got up to 10% RTT reductions.
Segment Routing and Egress Peer Engineering
SR represents any topological entities as segments implemented as MPLS labels in SR-MPLS or IPv6 addresses in SRv6. BGP-EPE (RFC 9087) applies this philosophy to eBGP peers.
Figure 1 illustrates an overview of BGP-EPE with SR-MPLS. The ASBR of AS64500 assigns BGP Peering segments (SIDs) for two peering routers. When the ASBR receives a packet with the SID, the ASBR strips the MPLS header and sends the packet to the peer corresponding to the SID. Therefore we can send arbitrary packets to given peers identified by BGP Peering SIDs regardless of BGP best paths.
Experiment at Interop Tokyo 2021
We conducted an experiment with BGP-EPE on the ShowNet (AS290) network at Interop Tokyo 2021 to measure the potential latency benefit of egress traffic engineering at a leaf of the Internet.
Interop Tokyo is a large annual exhibition of network technologies in Japan, and ShowNet is an ephemeral event network built at Interop Tokyo. ShowNet demonstrates new networking technologies and conducts interoperability tests while providing network connectivity for exhibitors and visitors.
In 2021, the ShowNet backbone was composed of SR-MPLS and SRv6 L3VPN. The three ASBRs of AS290 — ASR9902 from Cisco Systems, MX204 from Juniper Networks, and NE8000-X4 from Huawei — are capable of SR and BGP-EPE.
The measurement was simple. We performed ping and traceroute from ShowNet to 2.6M IPv4 addresses via 101 eBGP peers. These sessions are spread over three transit ASes and 43 peer ASes who agreed to join the experiment.
Alternate vs best paths — which wins?
At the end of the exhibition, we obtained greater than or equal to 10 RTTs for 2.3M IPv4 addresses via each eBGP peer. From this data, we revealed the potential latency improvement via non best paths using BGP-EPE.
Figure 3 shows the histogram of per-path minimum RTTs to the probed destinations, via the best paths or alternate paths. The y-axis is the number of BGP prefixes that include target addresses. As shown, the alternate paths provide shorter latency than the BGP best paths.
Figure 4 shows the rate of RTT improvement via alternate paths to RTT from the best paths for each prefix. About 23% of the prefixes have no improvement (x = 0), which means that the best paths have the minimum RTTs for those prefixes. Conversely, 77% of the prefixes have alternate paths of shorter latency than the best paths. The prefixes between 23% and 84% got up to 10% RTT reductions.
The result where latency for 77% of the prefixes can be improved is surprising. We then found a reason behind the extensive latency improvement — the imbalance of egress ASes, the first hop ASes on the paths.
Figure 5 shows the percentage of egress ASes on the best and alternate paths. One transit occupies 88% of egress ASes on the best paths while it is much less present on the alternate paths.
Shared-cost peers also exist as egress ASes, but their presence is very small compared with the number of paths through the transit ASes. Thus, they are not visible in Figure 5.
The imbalance shown in Figure 5 is due to the deployment process of ShowNet. Older paths from Transit A survived in accordance with the best path selection algorithm.
To sum up, we found that egress traffic engineering can improve latency. The gain depends on the connectivity of the different neighbors, which we found differs significantly.
See our paper ‘A First Measurement with BGP Egress Peer Engineering‘ at PAM 2022 for further details on our study.
Contributor: Cristel Pelsser is a professor at the University of Strasbourg where she performs research in Internet routing, measurements and security
Ryo Nakamura is a research associate at the University of Tokyo and a network operator of AS2500, AS2501, and Interop Tokyo ShowNet.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.