Communication networks grow increasingly complex as a result of rising application demands on network bandwidth and availability. To satisfy demands, network operators develop and deploy improved hardware, software, and traffic engineering solutions.
One common traffic engineering solution is traffic load balancing, where a router splits traffic across multiple next-hops (links) when it has multiple routes to an IP prefix. Operators can configure load balancing by, for example, enabling Equal-Cost Multi-Path (ECMP), available on most routers. Figure 1 provides a real example, where router R1 splits traffic between routers R2 and R3.
Like routers that do not send ICMP Time Exceeded messages, or that encapsulate packets using MPLS, routers that perform load balancing complicate understanding what routes packets traverse from a source towards a destination, which can in turn affect tasks such as management and troubleshooting.
Figure 1: Router R1 load balances traffic to destination prefix P between routers R2 and R3.
Routers that perform load balancing choose among multiple next-hop routers when forwarding packets. In Figure 1, router R1 chooses among R2 and R3. In general, the next-hop is chosen by computing a hash over a subset of fields in each packet’s IP header. Hashing different subsets of IP header fields allow control of a trade-off between the granularity of load balancing and impact on traffic. Below we describe common load balancing configurations, how well they can load balance traffic, and their possible impact on traffic.
- Per-application: Hashes source and destination ports (but not addresses) to compute next-hops.
- Granularity: May split traffic unevenly; however, usually deployed to forward traffic to next-hops that provide performance tailored to applications’ requirements regardless of traffic split ratios.
- Impact on traffic: All traffic from an application will take the same path and experience similar performance, but different applications may experience a different performance.
- Per-destination: Hashes source and destination IP addresses to compute next-hops.
- Granularity: May lead to occasional uneven traffic split if source-destination pairs exchanging a lot of traffic hash to the same next-hop.
- Impact on traffic: All packets from a source to a destination will take the same path and experience similar performances.
- Per-flow: Hashes source and destination IP addresses and ports; may consider the IPv6 flow label field.
- Granularity: More even traffic split compared with per-destination since the hash function considers more fields in the packet header.
- Impact on traffic: Packets belonging to the same connection will take the same path and experience similar performance, but performance may vary across connections.
- Per-packet: Next-hop is chosen at random.
- Granularity: Allows for perfectly even traffic-split ratios.
- Impact on traffic: Packets belonging to the same connection may take different paths and be delivered out of order, degrading TCP performance.
We have developed mtraceroute, a modified version of traceroute that identifies IPv6 routers that perform load balancing. Like traceroute, our tool identifies routers along a path, but sends additional packets, systematically varying select fields in the IPv6 header, to identify routers that perform load balancing and, if so, which IPv6 header fields they use to choose next-hops.
In our research work, recently presented at the Passive and Active Measurement Conference, we used our tool to characterize the extent and characteristics of IPv6 traffic load balancing. We measured 51,927 paths from 12 vantage points spread across seven countries. We observed that IPv6 load balancing is widespread, although less so than previously reported for IPv4. Per-destination and per-flow load balancing, which prevent packet reordering and do not affect TCP connections, are the most frequently observed types of load balancing.
Interestingly, we found that only a small fraction of IPv6 routers consider the IPv6 Flow Label field when choosing next hops; routers relied mostly on port numbers for per-flow load balancing. We found per-application load balancing to be rare, and usually located near the edge.
Finally, we did observe some instances of IPv6 per-packet load balancing, which can degrade TCP performance.
As future work, we plan to repeat our measurements from a more diverse set of vantage points (as more cloud providers add support for IPv6), and try to partner with cloud providers to obtain resources for a continued mapping of load balancing across the Internet.
Italo Cunha is an assistant professor at Universidade Federal de Minas Gerais, Brazil. His research focuses on improving network performance and reliability.
Article co-written by Rafael Almeida, a first-year MSc candidate at Universidade Federal de Minas Gerais, whose research focuses on computer networks and Internet measurement. Rafael is currently studying how traffic load balancing is performed on the IPv6 Internet. He maintains mtraceroute, an open-source version of traceroute capable of identifying routers that perform traffic load balancing.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.