Measuring BGP in 2023 — Have we reached peak IPv4?

At the start of each year, I report on the behaviour of the Internet’s inter-domain routing system over the previous 12 months, examining some metrics from the routing system that can show the essential shape and behaviour of the underlying interconnection fabric of the Internet.

The year 2023 marks a significant point in the evolution of the Internet where the strong growth numbers that were a constant feature of the past thirty years are simply not present in the data. Not only is the Internet’s growth slowing down significantly, but in the IPv4 network it appears to be shrinking, which is unprecedented in the brief history of the Internet to date.

Have we got to the point of market saturation and there is no more demand capacity to fuel further growth? Is the Internet losing its investment appeal along with so many other signals of investor disillusion over outlandish growth predictions in technology-based services? Or has the massive transition into Content Distribution Networks (CDNs) for digital services meant that there is a declining demand for the traditional form of content distribution over on-demand network access?

Let’s take a look at the BGP view of 2023 and see how these larger technical and economic considerations are reflected in the behaviour of the Internet’s inter-domain routing system.

The first metric of interest is the size of the routing tables. Each router needs to store a local database of all prefixes announced by each routing peer. In addition, conventional routing design places a complete set of ‘best’ paths into each line card and performs a lookup into this forwarding data structure for each packet. This represents an extremely challenging silicon design problem. The larger the routing search space, the more challenging the problem!

There is also the consideration of the overall stability of the system. Processing a routing update requires several lookups into local data structures as well as local processing steps. Each router has a finite capacity to process updates, and once the update rate exceeds this local processing capability, then the router will start to queue up unprocessed updates.

In the worst case, the router will start to lag in real-time, so that the information a BGP speaker is propagating reflects a past local topology, not necessarily the current local topology. If this lag continues, then at some point, unprocessed updates may be dropped from the queue. BGP has no inherent periodic refresh capability, so when information is dropped, then the router and its neighbors fall out of sync with the network topology.

At its most benign, the router will advertise ‘ghost’ routes where the prefix is no longer reachable, yet the out-of-sync router will continue to advertise reachability. At its worst, the router will set up a loop condition and as traffic enters the loop it will continue to circulate through the loop until the packet’s TTL expires. This may cause saturation of the underlying transmission system and trigger further outages which, in turn, may add to the routing load.

The two critical metrics we are interested in are the size of the routing space and its level of updates or churn. Here we will look at the first of these metrics, the size of the routing space, and the changes that occurred through 2023, and use this data to extrapolate forward and look at five-year projections for the size of the routing table in both IPv4 and IPv6.

The BGP measurement environment

In trying to analyse long baseline data series the ideal approach is to keep as much of the local data-gathering environment as stable as possible. In this way, the changes that occur in the collected data reflect changes in the larger environment, as distinct from changes in the local configuration of the data collection equipment.

The measurement point being used here is a BGP speaker configured within AS131072. This Autonomous System (AS) generates no traffic and originates no routes in BGP. It’s a passive measurement point that has been logging all received BGP updates since 2007. The router is fed with a default-free eBGP feed from AS4608, which is the APNIC network located in Australia, and AS4777, which is the APNIC network located in Japan, for both IPv4 and IPv6 routes.

There is also no internal routing (iBGP) component in this measurement setup. While it has been asserted at various times that iBGP is a major contributor to BGP scalability concerns in BGP, the consideration here in trying to measure this assertion is that there is no ‘standard’ iBGP configuration, as each network has its own rather unique configuration of Route Reflectors and iBGP peers. This makes it hard to generate a ‘typical’ iBGP load profile, let alone analyse the general trends in iBGP update loads over time.

In this study, the scope of attention is limited to a simple eBGP configuration that is likely to be found as a ‘stub’ AS at the edge of the Internet. This AS is not an upstream for any third party, it has no transit role and does not have a large set of BGP peers. It’s a simple view of the routing world that I see when I sit at the edge of the Internet. Like all BGP views, it is totally unique to this network, and every other network will see a slightly different Internet with different metrics. However, the behaviour seen by this stub network at the edge of the Internet is probably similar to most other stub networks at the edge of the Internet. While the fine details may differ, the overall picture is probably much the same. This BGP view is both unique and typical at the same time.

The IPv4 routing table

Measurements of the size of the routing table have been taken regularly since the start of 1988, although regular highly detailed snapshots of the routing system only date back to early 1994. Figure 1 shows a rather unique picture of the size of the routing table, as seen by all the peers of the Route Views route collector on an hourly basis.

Several broader events are visible in the history of the routing table, such as the bursting of the Internet bubble in 2001, the impact of the global financial crisis in 2009 and the lingering effects of the COVID-related shutdowns in late 2020. What is perhaps surprising is one ongoing event that is not visible in this plot.

Since 2011, the supply of IPv4 addresses has been progressively constrained as the free address pools of the various Regional Internet Registries (RIRs) have been exhausted. Yet there is no visible impact on the rate of growth of the number of announced prefixes in the global routing system since 2011. In terms of the size of the routing table, it’s as if the exhaustion of IPv4 addresses has not happened at all for the ensuing decade.

It is only by 2021 that we see some tapering of the growth of the size of the IPv4 routing table. BGP is not just a reachability protocol. Network operators can manipulate traffic paths using selective advertisement of more specific addresses, allowing BGP to be used as a traffic engineering tool. These more specific advertisements often have a restricted propagation.

This is evident in Figure 1, where the BGP routing table counts from both the Route Views peers and the peers of the RIPE NCC’s Routing Information Service (RIS) are combined. There is not a single plot in this figure where each BGP speak sees essentially the same network. There is a variance across the various peers of these route collectors, which is around 50,000 routes.

We can zoom in to look at just 2023, taking the same collection of Route Views and RIS BGP peers (Figure 2). The major discontinuity in this period occurred on 22 February 2023, where AS9121 withdrew 5,279 prefixes and AS47331 withdrew 2,427 prefixes. Both of these networks are operated by Türk Telecom and the withdrawals reflect the withdrawal of more specific routes.

Both Figures 1 and 2 illustrate an important principle in BGP, that there is no single authoritative view of the Internet’s inter-domain routing table, as all views are relative to the perspective of each BGP speaker.

These figures also illustrate that, at times, the cause of changes in routing is not necessarily a change at the point of origination of the route, which would be visible to all BGP speakers across the entire Internet, but it may well be a change in transit arrangements within the interior of the network that may expose, or hide, collections of routes. For example, a change in the number of advertised routes on 26 June was visible only for around one-half of the networks that feed these route collectors.

The next collection of plots (Figures 3 to 12) contains some of the vital characteristics of the IPv4 BGP network from the start of 2016 to the end of 2023, using the routing data collected from AS131072.

Figure 3 shows the total number of routes in the routing table over this period. This is no longer a classic ‘up and to the right’ Internet trajectory, and there was a significant route withdrawal event early in the year (withdrawal of more specifics by Türk Telecom in February 2023). It seems that the growth of the Internet, in terms of the size of the IPv4 routing table, is slowing down to the point that we may see the network shrink in size in 2024. We’ll return to this data when we look at projections for the size of the BGP FIB table in the coming years.

The eight-year period since the start of 2016 has seen the span of addresses advertised in the routing system slowing down until the start of 2021 (Figure 4). In 2021, we saw several large address blocks being advertised in the routing system by agencies associated with the US Department of Defence. Across 2022 and 2023, the total span of advertised IPv4 addresses has declined.

In terms of more specific advertisements and covering aggregate advertisements (Figure 5), the number of covering aggregate announcements increased across 2023, and the number of more specifics has reduced in size. Looking at the ratio of these two counts, the ratio has declined over 2023 (Figure 6). The average prefix size is now somewhat smaller than a /20 (Figure 7). Prefixes sizes of /24, /23 and /22 in total now account for 80% of the entire IPv4 routing table (Figure 8).

The topology of the network has remained relatively consistent, with the growth of the Internet being seen as a relatively static density of interconnectivity. The average AS-PATH length has been constant since mid-2020. (Figure 9).

Further evidence of the slowing of IPv4 Internet growth can be seen in the AS count (Figure 10). The growth of the AS count started to decline in late 2020 and continued to decline in the ensuing years. This is a likely signal of network saturation in many markets. The number of transit networks was held constant across 2022, which appears to be a related signal of market saturation (Figure 11).

The year-by-year summary of the IPv4 BGP network (as seen by AS131072) over the 2020 to 2023 period is shown in Table 1.

Table 1 — IPv4 BGP table growth profile.

In terms of advertised prefixes, the size of the routing table grew by some 3,000 entries or just 0.3%. The number of root prefixes increased by 12,000 entries, while the number of more specific routes decreased by approximately 9,000 entries. If the count of more specific entries is related to the requirements for traffic engineering and load balancing on transit paths within the network, then this declining number points to a drop in the importance of traffic engineering in the IPv4 transit network.

The total span of advertised addresses declined by some 18M IPv4 /32s across the year, while the number of routed Stub AS numbers (ASNs) (edge networks) grew by 1.4% in 2023, and the total number of ASes visible in the IPv4 network grew by just 200 ASes across 2023, or 1.5%.

Let’s examine the 10 networks that had the highest net withdrawal count for the year, and the highest net growth (Table 2).

Table 2 — IPv4 advertised prefix changes, top 10 ASes.

It appears that many of the withdrawals are more specifics rather than covering aggregates. In the case of growth networks, the top-ranked network for 2023, AS18403, FPT Telecom in Viet Nam has some 820 ‘root’ announcements, and the remaining 3,727 announcements are more specifics. A similar picture exists for AS16509, Amazon O2, where a net reduction in the advertised prefix count of some 3,885 entries could be affected by dispensing many more specific /24s.

These metrics show that the drivers for growth in the IPv4 BGP network have largely dissipated across 2023. We are likely seeing a number of factors at play behind this major change in the IPv4 network:

•  The first is the saturation of many Internet markets across the globe so that the amount of ‘green field’ expansion is far lower than, say, a decade ago.

• Secondly, we are seeing considerable concentration on the service market, where the level of utilization of addresses is vastly greater by both content and service publishers and end clients. The service and client numbers may be growing, but that does not necessarily imply the use of more IPv4 addresses or more routing table entries. Network Address Translators (NATs) are fully entrenched in the IPv4 world, and in the service provider market, volume economics implies that few larger providers are more efficient than a greater number of smaller providers.

• Thirdly, this concentration in the service market has been accompanied by further consolidation in the access market, particularly in mobile access networks. This consolidation of client access networks creates greater efficiencies in shared address solutions.

• The continued deployment of IPv6 cannot be ignored. Within the 10 economies with the largest span of advertised addresses (collectively, these 10 economies advertise 74% of the span of advertised IPv4 addresses) seven of these economies are also in the 10 economies with the largest span of advertised IPv6 addresses (collectively, these same 10 economies advertise 76% of the span of advertised IPv6 addresses). Looking at just these seven economies, namely the United States, China, Japan, Germany, the United Kingdom, Brazil, and Canada, they advertise 76% of the entire advertised IPv6 address span and 65% of the advertised IPv4 address span. The level of IPv6 use in six of these seven economies has a greater use of IPv6 deployment than the total global deployment level of 37% of end users, while the deployment levels in China is slightly below this global average.

• Finally, I suspect that the most significant dissipating factor has been the inexorable rise of private CDNs. The conventional drivers for growth in the routing table were based on the product of an increasing number of networks with discrete routing policies and the need for these networks to balance incoming traffic across multiple network paths (traffic engineering using more specifics). The rapid uptake of CDNs by service platforms has meant a significantly reduced level of dependence on BGP transit services to deliver content to users. The public part of the Internet is rapidly shrinking to the last-mile access network, which connects directly to several CDNs. The transit part of the Internet is just not so critical these days. It is unlikely that we will move away from the extensive use of CDNs in the foreseeable future. If anything, their level of use within the Internet’s service portfolio will increase further.

In summary, the IPv4 network may possibly have peaked in 2023. In various parts of the network, the number of IPv4 entries in the default-free zone is between 920,000 and 960,000 entries. The net growth in 2023 was some 3,000 new entries, down from 34,000 in 2022 and 53,000 in 2021. A net increase of 1,100 new ASNs was seen in the IPv4 network across the year, compared to 1,599 in 2022. The network is certainly slowing down at present, and the current open question is whether this growth will reverse in 2024 and begin to decline in size in the coming years.

The IPv6 BGP table data

A similar exercise has been undertaken for IPv6 routing data. As with the IPv4 network, there is diversity in the number of IPv6 routes seen at various vantage points, as shown when looking at the prefix counts advertised by all the peers of Route Views (Figure 13).

There are several distinct phases in the growth trends that are visible in the history of the IPv6 routing table. The period between 2004 and mid-2018 could be modelled by an exponential growth function with a doubling interval of three years. The period from mid-2019 to mid-2021 saw an increase in the growth function, where the doubling interval fell to slightly under two years. However, from mid-2021 the growth pattern has changed again and is now best modelled by a linear growth model with a growth rate of some 25,000 additional route entries per year, or an average of 68 new routing entries per day.

A more detailed look at the most recent four years incorporating both Route Views and RIS data (Figure 14) shows some increasing diversity between various BGP views as to what constitutes the ‘complete’ IPv6 route set. The variance at the end of 2023 now spans some 25,000 prefix advertisements.

Figure 15 shows a detailed view of the routing table as seen by each of the Route Views and RIS peers across 2023.

The comparable profile figures for the IPv6 Internet are shown in Figures 16 through 25.

The growth of the IPv6 routing table has been relatively steady across 2022, growing by some 30,000 routes across 2023 (Figure 16). For the first half of 2023, the average daily growth was an additional 100 to 120 new entries per day, declining to some 50 to 70 new entries in the latter half of the year. Routing advertisements of /48s are the most prevalent prefix size in the IPv6 routing table at the end of 2023, and some 46% of all prefixes are /48s. A total of 75% of the IPv6 table entries comprise /48, /32, /44, and /40 prefixes (Figure 21).

RIR allocations of IPv6 addresses show a different pattern, with 74% of the 64,008 IPv6 address allocations recorded in the RIR registries being either a /32 (48%) or a /29 (25%). Only 20% of allocations are a /48. It’s evident that there is no clear correlation between an IPv6 address allocation prefix size (as used by the address registries in the address allocation process) and the advertised address prefix size. Many IPv6 address holders do not advertise their entire allocated IPv6 address prefix in a single routing advertisement.

Why is the IPv6 routing table being fragmented so extensively? The conventional response is that this is due to using more specific route entries to perform traffic engineering. Another possible reason is the use of more specifics to counter efforts of route hijacking. This latter rationale also has issues, given that it appears that most networks appear to accept a /64 prefix, and the disaggregated prefix is typically a /48, so as a countermeasure for more specific route hijacks, advertising /48s may not be all that effective.

This brings up the related topic of the minimum accepted route object size.

The common convention in IPv4 is that a /24 prefix advertisement is the smallest address block that will propagate across the entire IPv4 Default-Free Zone (DFZ). More complex minimum size rules have largely fallen into disuse as address trading appears to have sliced up many of the larger address blocks into smaller sizes. If a /24 is the minimum accepted route prefix size in IPv4, what is the comparable size in IPv6? There appears to be no common consensus position here, and the default action of many network operators appears to use no minimum size filter at all.

In theory that would imply that a /128 route object would be accepted across the entire IPv6 DFZ, but a more pragmatic observation is that a /32 would be assuredly accepted by all networks, and it appears that many network operators believe that a /48 is also generally accepted. Given that a /48 is the most common prefix size in today’s IPv6 network this view appears to be widespread. However, we also see prefixes smaller in size than a /48 in the routing table with /49, /52, /56 and /64 prefixes present in the IPv6 BGP routing table. Slightly more than 1% of all advertised prefixes are more specific than a /48.

The summary of the IPv6 BGP routing table profile for the period 2020 through to the start of 2024 is shown in Table 3. The IPv6 network growth rate in 2023 has been on par with the previous year, with a 7% growth in networks, and a 17% growth in route entries.

Table 3 — IPv6 BGP table growth profile.

The pressures for further expansion on the IPv6 network appear to be more particular to each market sector and region, rather than some form of more general imperative. Where there is at present scant IPv6 adoption, as is the case in most of Africa, the Middle East, Eastern and Southern Europe, and the western part of Latin America, there is no apparent sense of urgency to make the shift (Figure 26). It would appear that the Internet market is largely a saturated one and the smaller pace of network growth in those regions, for the moment, can be adequately accommodated in the continued use of IPv4 NATs.

The predictions

What can this data from 2023 tell us in terms of projections of the BGP’s future in terms of BGP table size?

Forecasting the IPv4 BGP table

Figure 27 shows the data set for BGP from January 2016 until December 2023. This plot also shows the fit of these most recent five years of data to various growth models. The first-order differential, or the rate of growth, of the BGP routing table is shown in Figure 28. The linear average rate of growth of the routing table appears to be falling slowly from 140 to 160 additional entries per day in 2016 to around 100 per day at the start of 2024.

There are a number of potential models to match this data. One model is to take the five-year average daily rate of change and apply this as a continuous model for the next five years. This is a ‘linear’ model and takes the current dynamics of the IPv4 Internet and makes the assumption that these dynamics will operate largely unchanged over the projection period. The second model is to look at the trend in the changes of rate of change and match this to a linear model.

If the first-order differential of a data series is a linear function, then the original data can be represented as a second-order polynomial. The final model used here is to model the log of the data series as a linear model and therefore derive an exponential model for the data series. The application of these three projection models to the original data series is shown in Figure 27, and the first-order differential of the data (the daily rate of change) is shown in Figure 28.

Figure 28 appears to support the supposition of the primary data series fitting a second-order polynomial model. The rate of change of the daily change to the count of IPv4 routing objects has been declining since 2020, where the range of daily change has fallen from a band of values between 100 to 200 new entries per day to a new range that sits in a band between 50 to 100 new entries per day. This model suggests that the overall rate of change will shift from positive to negative in 2026.

The five-year prediction models for all three forms (linear, polynomial and exponential) are shown in Figure 29.

The projections of the linear and polynomial best-fit models are shown in Table 4 (the resumption of an exponential growth model appears to be highly unlikely in this late phase of the IPv4 network).

Table 4 — IPv4 BGP table size prediction.

The linear projection model is somewhat unlikely in this context. The drivers for the continued growth of the IPv4 network do not appear to be clear, so the projection of continued growth of the number of IPv4 FIB entries with a net gain of 43,000 is somewhat unrealistic.

The O(2) polynomial projection model predicts that this first-order differential will reach the zero point by 2026 and continue to decline. It must be stressed it’s just a mathematical model that fits the recent data, and nothing more. The dominant factor that is driving the prediction of a peak size in 2025 is the large number of more specific entries that were withdrawn by AS9121 and AS473331 at the start of 2023. If one were to modify the primary data series and remove this route withdrawal event from the series then the resulting picture is still a declining growth rate, but from a higher base and at a lower rate. This would push the peak point out to late 2027 or early 2028, but not change the overall pattern of a shift from growth to decline in the coming years.

Given that the last ‘normal’ year of supply of available IPv4 addresses to fuel continued growth in the IPv4 Internet was now over a decade ago in 2010, perhaps the more relevant question is — why has the growth of the IPv4 routing table persisted with such regularity in the ensuing decade?

It should be remembered that a dual-stack Internet is not the objective at this time of transitioning the Internet to IPv6. The ultimate objective of the entire transition process is to support an IPv6-only network. An important part of the process is the protocol negotiation strategy used by dual-stack applications, where IPv6 is the preferred protocol wherever reasonably possible. In a world of ubiquitous dual-stack deployment, all applications will prefer to use IPv6, and the expectation is that in such a world the use of IPv4 would rapidly plummet.

The challenge for the past decade or more has been in attempting to predict when in time that tipping point that causes demand for IPv4 to plummet may occur. The current data suggests that this point of inflexion for the IPv4 network may be imminent.

Forecasting the IPv6 BGP table

The same technique can be used for the IPv6 routing table. Figure 30 shows the data set for BGP from January 2016 until January 2024.

The first-order differential, or the rate of growth of the IPv6 BGP routing table is shown in Figure 31. The number of additional routing entries has grown from 10 new entries per day at the start of 2012 to a peak of some 230 new entries per day in May 2021.

The first half of 2023 had a steady period of increase in the growth of the daily count of new entries from around 100 per day to 150 per day, while the second half of 2023 saw a decline in this rate of growth to 50 new entries per day by the end of the year (Figure 31).

There are two visible trends in the IPv6 data. The five-year trend can be fitted against an exponential growth model with a doubling interval of a little under three years. The more recent two-year trend can be fitted against a linear growth model. The fit of a linear projection and an exponential model of projected IPv6 table size is shown in Figure 32.

The projections for the IPv6 table size are shown in Table 5.

 

Table 5 — IPv6 BGP table size prediction.

The linear and exponential projections in Table 5 provide a reasonable estimate of the low and high bounds of the growth of the IPv6 BGP routing table in the coming years.

If IPv6 continues to grow exponentially over the next five years, doubling every 30 months or so, then the size of the IPv6 routing table will be a little over one million entries at the start of 2030. The data from the previous two years suggests that such a level of growth is unlikely, and a linear growth model is a closer fit to the recent past, and an average growth rate of 23,000 new entries per year is a better fit for this recent data.

What is also unknown is the extent to which the slowing growth rate for IPv6 in the past 24 months is related to the COVID-19 pandemic, and potentially the economic side effects of the turmoil in global energy markets due to the situation in Ukraine. The communications market is not immune from such global effects.

Conclusion

These predictions for the routing system are highly uncertain. The correlation between network deployments and routing advertisements has been disrupted by the hiatus in the supply of IPv4 addresses, causing more recent deployments to make extensive use of various forms of address-sharing technologies, and making fundamental alterations to the architecture of the service model of the Internet.

While a number of access providers and service platforms have made significant progress in public IPv6 deployments for their respective customers, the majority of the Internet user base (slightly under two-thirds of the Internet’s user base) is still exclusively using IPv4 as of the end of 2022 (Figure 33).

The predictions as to the future profile of the routing environment for IPv4 and IPv6 that use extrapolation from historical data can only go so far in providing a coherent picture for the near-term future. As well as the technical issues relating to the evolution of IP technology and the IPv6 transition, there are also broader factors such as the state of the global communications economy and the larger global economy. Investment in communications infrastructure, as with most other forms of infrastructure investment is not generally a short-term proposition.

The major benefits tend to be realized in increased efficiency of economic production, rather than short-term windfall gains from infrastructure investment. This means that short-term expedient measures, such as a response to a global pandemic or a rapid escalation of energy prices due to regional conflict, can interrupt infrastructure investment programs.

The question behind the recent slowing of the growth in both the IPv4 and IPv6 aspects of the Internet’s routing space is whether this slowdown is due to market saturation in the case of IPv4, or a dissipation of collective market impetus in the case of IPv6, or an interruption due to these short-term exogenous market factors. In the latter case, we would expect growth to resume once more when the current global market conditions dissipate, while an underlying condition of market saturation is a more permanent state.

If the concern is that the routing system is growing at a rate that is faster than our collective ability to throw available technology at it, then there is no serious cause for alarm in the current trends of growth in the routing system. There is no evidence of the imminent collapse of BGP. Far from it!

However, the size of the inter-domain routing table is only one-half of the story. The stability of the routing system is also very important, and to complete this look at the routing system in 2023 we will also need to look at the dynamic behaviour of the routing system. The profile of BGP update churn in 2023 is a topic we’ll look at in detail in the next post, which looks at the larger picture of addressing and routing across 2023.

Rate this article

---

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.