On 12 August 2014, the Internet passed a milestone, but it wasn’t necessarily one to celebrate.
The IPv4 Internet routing table exceeded 512,000 BGP routes. Network operators who were using older kits and hadn’t adequately prepared ran into trouble as they pushed up against the maximum number of routes in their BGP table settings.
Of course, it wasn’t the end of the world for the Internet. But there were some unfortunate consequences, including a route leak of 22,000 prefixes in the Default-Free Zone (DFZ).
The event also triggered a 512k Forwarding Information Base (FIB) limit exhaustion on commonly used router models, and network operators around the world experienced incidents related to router reset and reachability.
History repeats itself?
The BGP table continues to grow. In the past year, IPv4 prefixes have grown at 6% and IPv6 at 33%. In January 2021, per the CIDR report, the table stood at around 861,418 IPv4 prefixes, and 109,398 for IPv6.
So, it’s worth asking: What are the next numbers that might trigger similar problems to the 2014 event?
Read more: BGP in 2020 – The BGP Table
The numbers to watch
Routers are configured with certain maximum sizes for the routing tables. There are various models of routers deployed in operators and end user networks. Some of these router models could have limits for IPv4 set at 1,048,576 routes, and for IPv6 the next likely limit stands at 131,072.
IPv4: 1024k = 1,048,576
IPv6: 128k = 131,072
Over the years, network equipment manufacturers have released different router models based on hardware architecture, silicon pricing, and targeting network use cases. Network equipment vendors release these models in market with well tested scale limits. The scale limits of router memory and software data structures are usually aligned to some well-known boundary numbers like 128k, 512k, 1024k, 2048k, and so on.
For routers that are expected to carry the full Internet routing table, its is expected that the network operator has chosen and deployed the router with the right capabilities, such as CPU, memory, and enough head room to cater for routing table growth.
|Version||CIDR Report |
(31st Jan 2021)
|Colt AS8220 |
(31st Jan 2021)
Table 1 — Key numbers for IPv4 and IPv6.
Based on current trends, the 128k IPv6 milestone is likely to be around the end of 2022. The next IPv4 key number is likely the 1024k milestone around the end of 2023.
Not every operator is the same
It’s not a simple matter of every network operator running into this problem. The internal tables of network operators can vary according to design. Key factors include:
- Topology — Native IP vs MPLS
- Interior Gateway Protocol (IGP) — ISIS/OSPF size
- Customer routes
- BGP address families — IPv4, IPv6 / IPv6 labelled unicast
There are a wide variety of scenarios that could play out when, or even before, the routing table hits those IPv6 and IPv4 milestones, such as:
- A large routing table leak (10k – 30k) could breach these special numbers ahead of time
- Network hardware deployed over the years could hit some limits with respect to 1024k and 128k. Software configuration and knobs could hit limits of 1024k or 128k
- There could be a significant event across the Internet if enough operators are affected
- A low impact event could occur on operator networks based on network deployment at different times
- Nothing might happen! Operators may have already managed to optimize ahead of time
At my company, Colt, we decided to check on the limits in our network devices. Our checks came mainly in two forms: Checking the hardware and silicon architecture limits on different router models, and checking the network OS configuration and standards.
Checking the network hardware
If all network operators used the latest hardware, with its higher memory and route memory limits, there wouldn’t be an issue to consider. But the reality is, for a variety of reasons, there are plenty of operators using older equipment.
The wide variety of routers available means there are a wide variety of architecture and router memory considerations.
Router hardware on the market today can be broadly classified into fixed and modular architecture. The routers come in different form factors and scale to fit network use cases. Different router models are deployed by operators in their network topology, based on these scale limit and business needs.
Operators need to understand their router hardware architecture and monitor the scale limits of different components. From a route scale growth perspective, a router has three different tables where IP address prefixes are stored, along with the next hop address:
- BGP Routing Information Base (BGP RIB)
- Routing Information Base (RIB)
- Forwarding Information Base (FIB)
The FIB is constructed from the main routing table that is used to forward packets to the router, identified by the next hop address. Older routers use regular RAM to store the FIB, some use Ternary Content Addressable Memory (TCAM).
The general recommendation would be to buy routers that can handle the BGP RIB, RIB and FIB with enough head room for future IPv4 and IPv6 scales. Unfortunately, it’s not always obvious how many prefixes a router can handle, especially if the TCAM is also used for Access Control Lists (ACLs) and Quality of Service (QoS) features. The operator needs to anticipate the scale needs for their router deployment and use cases.
If the network operator is not able to remove older router hardware and move to newer models, then they may need to explore options to optimize RIB and FIB usage. It’s prudent to take advice and guidelines from router manufacturers on temporary options to adjust the TCAM size and control routes from RIB to FIB, and so on. These can be considered to be ‘patches’ that protect routers from failing.
Checking network operating system configurations
Operators should review and verify their network operating system configuration standards. Protocol feature configurations and knob behaviours may look similar, but have subtle differences across operating systems. Network operating system behaviours also change with software version upgrades.
As an example, ‘maximum prefix limit’ is attached to BGP sessions to protect against route leaks. Operators usually apply BGP maximum prefix limits on eBGP (peering/customers/external). Maximum prefix limit may be not widely deployed on iBGP sessions for IPv6 and IPv6, since the protection is already covered at the edge of the network.
A subtle difference across network operating systems that may be worth noting, is that Cisco IOS-XR applies a default value if the maximum prefix is not applied. The default value for widely used platforms running IOS-XR for IPv4 unicast is 1,048,576. For IPv6-labelled unicast the default is 131,072.
This shows that operators also need to review their iBGP configurations. They may need to adjust the maximum prefix on some network operating systems ahead of time, to avoid resetting iBGP sessions.
Advice for operators
We suggest that operators review the hardware and software scale limits of all running platforms, processors, line cards, and the RIB and FIB limits. Talk to your vendor to validate route optimization knobs that could be applicable to your network environment. The community of network builders, operators, and admins needs to look at diligently optimizing their networks with a continuous and iterative approach. This is crucial to a stable, reliable, and secure Internet.
To see the full APRICOT presentation, check out the embedded video of the APOPS 2 session below, or follow this link.
Danny Pinto is Associate Director, Packet Networks Engineering at Colt Technology Services
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.