Measuring the potential benefit of egress traffic engineering with Segment Routing

By on 10 Mar 2022

Category: Tech matters

Tags: , , , ,

Blog home

Photo: Nic McPhee, Flickr.

Border Gateway Protocol (BGP) best paths on which the Internet relies are not always the ‘best’ when it comes to bandwidth and latency. This is due to several factors including BGP not considering congestion on links and AS-PATHs using prepending no longer reflect closeness.

As such, engineering egress traffic toward neighboring Autonomous Systems (ASes) is a constant challenge for network operators seeking better paths.

To begin with, the potential benefit of egress traffic engineering is unclear. Measuring the performance of a non-best path to a destination requires preferring the non-best path. However, using local preference to do this impacts all user traffic to the destination.

In an effort to improve our understanding of this, we experimented with Segment Routing (SR) based BGP Egress Peer Engineering (BGP-EPE). We measured non-best paths without impacting user traffic by relying on encapsulation to a chosen exit. We showed that 77% of prefixes have alternate paths of shorter latency than best paths and about 60% of prefixes got up to 10% RTT reductions.

Segment Routing and Egress Peer Engineering

SR represents any topological entities as segments implemented as MPLS labels in SR-MPLS or IPv6 addresses in SRv6. BGP-EPE (RFC 9087) applies this philosophy to eBGP peers.

Figure 1 illustrates an overview of BGP-EPE with SR-MPLS. The ASBR of AS64500 assigns BGP Peering segments (SIDs) for two peering routers. When the ASBR receives a packet with the SID, the ASBR strips the MPLS header and sends the packet to the peer corresponding to the SID. Therefore we can send arbitrary packets to given peers identified by BGP Peering SIDs regardless of BGP best paths.

Infographic showing overview of BGP-EPE with SR-MPLS.
Figure 1 — Overview of BGP-EPE with SR-MPLS.

Experiment at Interop Tokyo 2021

We conducted an experiment with BGP-EPE on the ShowNet (AS290) network at Interop Tokyo 2021 to measure the potential latency benefit of egress traffic engineering at a leaf of the Internet.

Interop Tokyo is a large annual exhibition of network technologies in Japan, and ShowNet is an ephemeral event network built at Interop Tokyo. ShowNet demonstrates new networking technologies and conducts interoperability tests while providing network connectivity for exhibitors and visitors.

In 2021, the ShowNet backbone was composed of SR-MPLS and SRv6 L3VPN. The three ASBRs of AS290 — ASR9902 from Cisco Systems, MX204 from Juniper Networks, and NE8000-X4 from Huawei — are capable of SR and BGP-EPE.

Photo of ShowNet booth at the Interop Tokyo Exhibition.
Figure 2a — ShowNet booth at the Interop Tokyo Exhibition.
Topology diagram of ShowNet in 2021.
Figure 2b — The topology diagram of ShowNet in 2021. The full-size version of the diagram is available for download. The ShowNet icons used in the diagram (and Figure 1) are also available under CC-BY-SA 4.0.

The measurement was simple. We performed ping and traceroute from ShowNet to 2.6M IPv4 addresses via 101 eBGP peers. These sessions are spread over three transit ASes and 43 peer ASes who agreed to join the experiment.

Alternate vs best paths — which wins?

At the end of the exhibition, we obtained greater than or equal to 10 RTTs for 2.3M IPv4 addresses via each eBGP peer. From this data, we revealed the potential latency improvement via non best paths using BGP-EPE.

Figure 3 shows the histogram of per-path minimum RTTs to the probed destinations, via the best paths or alternate paths. The y-axis is the number of BGP prefixes that include target addresses. As shown, the alternate paths provide shorter latency than the BGP best paths.

Histogram of RTTs via the best paths and alternate paths.
Figure 3 — Histogram of RTTs via the best paths and alternate paths. RTTs are summarized by the prefixes present in the BGP table of our measurement point.

Figure 4 shows the rate of RTT improvement via alternate paths to RTT from the best paths for each prefix. About 23% of the prefixes have no improvement (x = 0), which means that the best paths have the minimum RTTs for those prefixes. Conversely, 77% of the prefixes have alternate paths of shorter latency than the best paths. The prefixes between 23% and 84% got up to 10% RTT reductions.

CDF of improved latency by the alternate paths.
Figure 4 — CDF of improved latency by the alternate paths.

The result where latency for 77% of the prefixes can be improved is surprising. We then found a reason behind the extensive latency improvement — the imbalance of egress ASes, the first hop ASes on the paths.

Figure 5 shows the percentage of egress ASes on the best and alternate paths. One transit occupies 88% of egress ASes on the best paths while it is much less present on the alternate paths.

Shared-cost peers also exist as egress ASes, but their presence is very small compared with the number of paths through the transit ASes. Thus, they are not visible in Figure 5.

Graph showing percentage of egress ASes on the best and alternate paths.
Figure 5 — Percentage of egress ASes on the best and alternate paths.

The imbalance shown in Figure 5 is due to the deployment process of ShowNet. Older paths from Transit A survived in accordance with the best path selection algorithm.

To sum up, we found that egress traffic engineering can improve latency. The gain depends on the connectivity of the different neighbors, which we found differs significantly.

See our paper ‘A First Measurement with BGP Egress Peer Engineering‘ at PAM 2022 for further details on our study.

Contributor: Cristel Pelsser is a professor at the University of Strasbourg where she performs research in Internet routing, measurements and security

Ryo Nakamura is a research associate at the University of Tokyo and a network operator of AS2500, AS2501, and Interop Tokyo ShowNet.

Rate this article

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.

Leave a Reply

Your email address will not be published.

Top