Accidental route leaks and hijacks have been business-as-usual ever since Yakov Rekhter and Kirk Lougheed sketched out the Border Gateway Protocol (BGP) on two napkins 35 years ago. It has taken all this time to realize that a very simple tweak, though very limited in its nature, could have prevented the vast majority of those incidents. That tweak is now defined in RFC 9234, which, as I said, is something we wish had existed back then.
But it’s never too late. The RFC was published three years ago, and the first implementations are now appearing. Quagga/FRR have already implemented it, and there’s a patch available for BIRD. The first commercial implementation has also arrived — HPE Juniper Networking supports RFC 9234 as of Junos OS Release 25.2R1. More vendors are likely to follow!
In its essence, RFC 9234 is simple. It’s directly derived from the model of relationships between two adjacent Autonomous Systems (ASes) in BGP, which in its nature can be either provider-customer, often being offered to the customer by their provider under a commercial agreement or peering, which is often settlement-free. And every network engineer who ever touched BGP knows that with respect to each BGP neighbor, your network can play one of the following roles:
- Provider — neighbor AS for which your AS provides full transit to the Internet and back.
- Customer — neighbor AS, which your network uses for full transit to the global network.
- Peer — neighbor AS, which offers you limited transit to their network and the networks of their customers and vice versa, to whom you offer transit to your network and the networks of all your customers only.
Or, in a nutshell:
In its original specification, BGP did not account for the roles an AS might have with respect to its neighbors. The standard left every AS free to define routing policies however they wished. In theory, any AS can be configured to provide transit to anyone else. Yet, even early experiments back in 1991 showed that once NSFNET was decommissioned in 1995, leaving BGP to operate solely on AS-PATH length would lead to an unmanageable situation. The need for routing controls gave rise to the concept of roles, as well as the well-known Tier 1/2/3 architecture of the global network — a structure that has shaped interconnection ever since, with some recent refinements, as excellently observed by Nina Hjorth Bargisen in her article on the RouteViews site
Still, nothing changes with respect to the roles of Customer, Peer, and Provider and the way we implement those. Traditionally, we needed complex routing policies on our BGP-speaking routers to implement those, such as Cisco route-map / route-policy, Juniper/Nokia policy-statement, and so on. In those policies, we had to ensure that:
- Provider — advertise the full routing table (including or excluding the default route) to the Customer, allowing access to ANY destination on the global Internet, while accepting IP prefixes owned by the Customer and Customer’s customers only.
- Customer — advertise their own routes and the routes of their customers to the Provider, while accepting ANY route (or all of them) from the Provider, as they wish.
- Peers — each of them advertises only their own routes — routes belonging to themselves and/or routes belonging to their customers, customers’ customers and so on — recursively all the way down the rabbit hole (a.k.a. ‘customer cone‘).
This is shown in Figure 2. AS B, together with its customers E and F, forms the ‘AS B Customer Cone’. Similarly, AS C with D and E forms another cone. As shown, AS E is a customer of both C and B, so it belongs to both cones. Hence, customer cones may overlap with each other, and that’s normal.
At the edges, the ‘leaf’ networks (D, E, F, and I) advertise only the IP prefixes originating in their ASes to their upstream providers (C, B, and H). In turn, C, B, and H advertise their entire customer cones to their own upstreams (A and G). Meanwhile, A and G, as well as B and H, maintain peering relationships with one another, exchanging only the routes from their respective cones.
Typically, implementing routing policies like this involves creating BGP import / export policies towards a BGP neighbor, where we implement the policies stated above. For instance, if we model the network shown in Figure 1 with one router for each network, the configuration of the router AS200 would look like Figure 3.
In the Internet Routing Registries (IRRs) such as the RIPE Database or RADB, these relationships are documented using RPSL in the following way:
aut-num: AS200
import: from AS100 accept AS100:CUSTOMERS
export: to AS100 announce AS200:CUSTOMERS
import: from AS1 accept ANY
export: to AS1 announce AS200:CUSTOMERS
import: from AS250 accept AS250
export: to AS250 announce ANYThe traditional way
As operators have been doing for decades, the more or less classical way of implementation of the above-defined rules is performed as follows:
- Assign a standard community value for each neighbor role (Customer, Peer, Provider).
- In BGP import policies, tag all routes received with the appropriate community role.
- In BGP export policies, advertise only routes tagged with the Customer community to Peers and Providers, while advertising everything to all neighbors with the Customer role.
For instance, in Junos, this would be implemented using standard BGP import/export policies. For simplicity, we’ll show only the export policies that implement the rules above, and only an excerpt of them. The import policies are symmetrical:
policy-options {
policy-statement PS-EBGP-CUSTOMER-OUT {
term BGP {
from {
protocol bgp;
prefix-list PL-CUSTOMER; # Containing all Customer prefies
}
then accept;
}
then reject;
}
policy-statement PS-EBGP-PEER-OUT {
term REJECT {
from community [ PEER PROVIDER ];
then reject;
}
term BGP {
from {
protocol bgp;
community CUSTOMER;
}
then accept;
}
then reject;
}
policy-statement PS-EBGP-PROVIDER-OUT {
term REJECT {
from community [ PEER PROVIDER ];
then reject;
}
term BGP {
from {
protocol bgp;
community CUSTOMER;
}
then accept;
}
then reject;
}
community CUSTOMER members 65000:0;
community PEER members 65000:1;
community PROVIDER members 65000:2;
}Although we’ve been doing this for years, mistakes still happen. Automation helps reduce errors, but urgent, manual changes to router configurations — often made under pressure during critical outages — can easily introduce problems. What if the wrong community is applied? What if the export policy isn’t referenced in the BGP neighbor configuration?
The example above is very simple. Now imagine a network with 2,000 BGP neighbors, each with its own pair of import/export policies, and each policy containing hundreds of terms. It’s easy for one term to be overlooked. Most of the time, this leads to unwanted route leaks.
For instance, suppose we forget to apply the export PS-EBGP-PEER-OUT policy in the BGP neighbor configuration between AS100 and AS200, instead of advertising only the routes of AS200 and AS250:
root@AS200> show configuration protocols bgp group PEER
type external;
import PS-EBGP-PEER-IN;
neighbor 100.100.3.1 {
peer-as 100;
}
root@AS200> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 3 Peers: 3 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
5 4 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
100.100.3.1 100 414 414 0 0 3:06:35 Establ
inet.0: 0/1/1/0
100.100.5.2 250 418 419 0 0 3:06:35 Establ
inet.0: 1/1/1/0
100.100.8.1 1 413 415 0 0 3:06:35 Establ
inet.0: 3/3/3/0
root@AS200> show route advertising-protocol bgp 100.100.3.1
inet.0: 17 destinations, 18 routes (17 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 150.150.0.0/16 Self 250 I
* 200.200.0.0/16 Self I We would have:
root@AS200> show route advertising-protocol bgp 100.100.3.1
inet.0: 17 destinations, 18 routes (17 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 1.1.0.0/16 Self 1 I
* 2.2.0.0/16 Self 1 2 I
* 3.3.0.0/16 Self 1 3 I
* 150.150.0.0/16 Self 250 I
* 200.200.0.0/16 Self I In other words, we are leaking AS1, AS2, and AS3 routes to AS100. Not good — and certainly not how it’s meant to work. If AS100 had a higher local preference for routes learned from a Peer than from a Provider, it would prefer to reach AS1, AS2, and AS3 via AS200 instead of using its direct links (see Figure 1). In this simple example, the mistake is easy to spot and correct, but what if the scenario were much more complex?
RFC 9234 — the safety pin
RFC 9234 offers a neat solution, the so-called Only-To-Customer (OTC) BGP Capability and Attribute. HPE Juniper Networking has recently implemented this, providing a simple way to prevent issues like the one described.
As mentioned earlier, RFC 9234 is straightforward — it defines the LOCAL ROLE of your AS with respect to each of your neighbors. While your policies and neighbor classes may vary, it’s important to emphasize that RFC 9234 roles are defined from the perspective of your own AS relative to the rest of the Internet. As shown in Figure 3 above:
- AS200 is a Customer of AS2
- AS200 is a Provider to AS250
- AS200 is a Peer with AS100
How do we use this? The basic usage is simple. Just go to your BGP configuration and add the otc-local-role knob:
set protocols bgp group TRANSIT description "AS200's Transit Providers"
# AS200 is a Customer of their TRANSIT providers, so ...
set protocols bgp group TRANSIT otc-local-role customer
set protocols bgp group TRANSIT neighbor 2.2.2.2 peer-as 2 # AS2
set protocols bgp group CUSTOMER description "AS200's own Customers"
# AS200 is a Provider for their CUSTOMERs, so:
set protocols bgp group CUSTOMER otc-local-role provider
set protocols bgp group CUSTOMER neighbor 150.150.0.2 peer-as 250 # AS250
set protocols bgp group PEER description "AS200's Settlement-Free Peers"
# AS200 is a Peer for their PEERs, so:
set protocols bgp group PEER otc-local-role peer
set protocols bgp group PEER neighbor 5.5.5.5 peer-as 100 # AS100And that’s it. You can configure this at the BGP global level ([edit protocols bgp]), group level ([edit protocols bgp group XXX]), or neighbor level ([edit protocols bgp group XXX neighbor a.b.c.d]), as documented in the Junos Reference for the otc-local-role knob.
Important note: Committing the configuration above will reset all BGP sessions. Changing the role of a group or neighbor or deleting the OTC role will reset all BGP sessions for that group or neighbor. Keep this in mind when applying changes.
Once you commit the configuration, even though you forgot the BGP export policy on AS200 towards AS100, you will notice that AS200 now advertises the routes correctly:
root@AS200> show configuration protocols bgp group PEER
type external;
import PS-EBGP-PEER-IN;
otc-local-role {
peer;
}
neighbor 100.100.3.1 {
peer-as 100;
}
root@AS200> show route advertising-protocol bgp 100.100.3.1
inet.0: 17 destinations, 18 routes (17 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 150.150.0.0/16 Self 250 I
* 200.200.0.0/16 Self ISo we fixed it, and no policy included! Yay!
However, even though this knob can simplify your routing policies, remember that it is not a Swiss-army knife for every scenario, nor a cure-all for poor routing architecture or design. It is certainly not intended to be used as the sole mitigation tool. Well-crafted routing policies, respecting the principles in RFC 8212 and MANRS, remain essential.
Think of this feature as a safety pin or ‘circuit breaker’, protecting you and your BGP neighbors in the same way that RCD breakers safeguard electrical installations and appliances in your home.
RFC 9234 Principle of Operation — in a Nutshell
Just like any other BGP extension, this one is also based on two mechanisms:
- BGP Capability — negotiated using BGP OPEN messages when a BGP session is established. If BGP Role Capability (Code: 9) is present in the BGP Capabilities vector of the OPEN message, this option is communicated to the peer and BGP Role Attribute is exchanged using UPDATE messages. Otherwise, it’s only used locally, within the local AS, without being advertised to or received from the remote neighbor.
- BGP Attribute — if capability negotiation is successful, BGP OTC Attribute (Code: 35) is exchanged with the peer. The purpose of this attribute is to enforce that once a route is sent to a Customer, a Peer, or a Route Server Client, it will subsequently go only to the Customers. This attribute contains only the AS Number (ASN) from which it is received.
How is that latter attribute used? Suppose AS200 had another customer — AS500. If no export policies are used and the BGP roles are correctly set on the AS200 router for AS500 (provider), the routes received from AS100 should be propagated to AS500, but not to AS2 or AS100. Let’s check the route 150.150.0.0/16 coming from AS100, and let’s disable the direct BGP session between AS200 and AS250:
root@AS200> show bgp summary
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
100.99.1.1 200 15 8 0 0 3:08 Establ
AS500.inet.0: 6/6/6/0
100.99.1.2 500 10 14 0 0 3:08 Establ
inet.0: 0/0/0/0
100.100.3.1 100 36 36 0 2 15:09 Establ
inet.0: 2/2/2/0
100.100.5.2 250 542 548 0 1 18 Idle
100.100.8.1 1 96 95 0 0 41:12 Establ
inet.0: 3/3/3/0
root@AS200> show route advertising-protocol bgp 100.99.1.2
inet.0: 20 destinations, 20 routes (20 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 1.1.0.0/16 Self 1 I
* 2.2.0.0/16 Self 1 2 I
* 3.3.0.0/16 Self 1 3 I
* 100.100.0.0/16 Self 100 I
* 150.150.0.0/16 Self 100 250 I
* 200.200.0.0/16 Self I
root@AS200> show route advertising-protocol bgp 100.100.8.1
inet.0: 20 destinations, 20 routes (20 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 200.200.0.0/16 Self ISo, 150.150.0.0/16 with AS_PATH: 100 250, is correctly advertised to AS500, while not being advertised to AS1, so this is fine.
Strict or loose?
As mentioned before, when a BGP session is established and roles are defined, they are exchanged in the OPEN messages using BGP OTC Capability. But what if one of the neighbors does not support this capability? As with any other BGP Capability, it will be ignored by the neighbor not supporting it. However, RFC 9234 Chapter 4.2, envisages a slightly different handling of this attribute:
- Loose mode — This is the default in most implementations. In this mode, a neighbor that does not understand the BGP OTC Capability will simply ignore it. The BGP session is still established, but the OTC Role configured on the supporting router is applied only locally. It is not propagated in BGP UPDATE messages to the neighbor. Even so, the local network still benefits. Routes from neighbors with an otc-local-role of customer or peer are correctly blocked from being advertised to providers (neighbors with the customer role). At the same time, these routes can still be advertised to customers — that is, to neighbors for whom the local router is the provider. Remember: The local router is always the reference point when designating roles
- Strict mode — if the BGP OTC Capability is not negotiated — DO NOT create the BGP session and handle it as an error, as per RFC 7606.
How does strict mode work? Let’s configure it on AS100 for the BGP session with AS200, and then remove the OTC role from the AS200 router, for example:
root@AS100# show protocols bgp group PEER
type external;
import PS-EBGP-PEER-IN;
export PS-STATIC-TO-BGP;
otc-local-role {
peer {
strict; ### Turn on the OTC Strict Mode !!!
}
}
neighbor 100.100.3.2 {
peer-as 200;
}
[edit]
root@AS200# show protocols bgp group PEER
type external;
import PS-EBGP-PEER-IN;
neighbor 100.100.3.1 {
peer-as 100;
}
[edit]
root@AS200#Now, let’s commit this configuration on both sides and check the syslog:
root@AS100> show log messages | match bgp | last 3
Aug 15 17:21:51 AS100 rpd[12593]: %DAEMON-4: bgp_process_role_cap:201: NOTIFICATION sent to 100.100.3.2 (External AS 200): code 2 (Open Message Error) subcode 11 (invalid)Policy-based OTC role override
Sometimes, setting a fixed role per BGP group or neighbor may be insufficient. Twenty years ago, in the traditional ISP environment, especially within Tier 1 carrier networks with well-structured and (mostly) clean BGP policies, this limitation was rarely an issue. But in the mobile and enterprise segments, it can create restrictions. For example, what if a neighbor is defined as a Peer, yet you still need to leak a route received from them to other peers or even transit providers? In mobile core networks, this situation is not unusual, it is often the norm.
Junos has a nice solution for that, too! You can override a role for one or more prefixes, routes having certain attributes (AS-PATH, community, and so on) or whatever you can match in the Routing Policy Match Conditions (‘from’ statement) and assign it a specific OTC Role within the Routing Policy Actions (“then” statement). As of Junos 25.1R2, there is an action for that:
root@AS100# set policy-options policy-statement MYPOLICY term OTC-ROLE-OVERRIDE then otc-local-role ?
Possible completions:
customer Customer role
peer Peer role
provider Provider role
route-server Route server role
route-server-client Route server client role
[edit]Let’s see this one in action now. Let AS200 advertise the route 200.200.5.0/24 to AS100, aside of the /16 route:
root@AS200> show route advertising-protocol bgp 100.100.3.1
inet.0: 21 destinations, 21 routes (21 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 200.200.0.0/16 Self I
* 200.200.5.0/24 Self ILet’s check what AS100 receives:
root@AS100> show route receive-protocol bgp 100.100.3.2
inet.0: 23 destinations, 28 routes (23 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 200.200.0.0/16 100.100.3.2 200 I
* 200.200.5.0/24 100.100.3.2 200 ISo far, so good. Now, let’s check what AS100 advertises to its providers, for example, AS1:
root@AS100> show route advertising-protocol bgp 100.100.6.1
inet.0: 23 destinations, 28 routes (23 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 100.100.0.0/16 Self I
* 150.150.0.0/16 Self 250 I That’s correct. Now, suppose AS100 needs to leak the prefix 200.200.5.0/24, which it received from its peer AS200, towards its upstream providers (AS1 and AS2). Since AS100 has assigned AS200 the Peer role, any routes received from AS200 are blocked from being propagated to other peers or providers, that is, to neighbors for which AS100’s local role is Peer or Customer. Simply adding an exception in the EBGP export policy toward AS1 and AS2, without overriding the OTC Local Role, will not work. The routes will still be filtered:
root@AS100# show policy-options policy-statement PS-EBGP-TRANSIT-OUT
term EXCEPTION {
from {
protocol bgp;
route-filter 200.200.5.0/24 exact;
}
then accept;
}
term BGP {
from {
protocol bgp;
community CUSTOMER;
}
then accept;
}
then reject;
root@AS100# run show route advertising-protocol bgp 100.100.6.1
inet.0: 23 destinations, 28 routes (23 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 100.100.0.0/16 Self I
* 150.150.0.0/16 Self 250 I
[edit]
root@AS100# run show route advertising-protocol bgp 100.100.7.1
inet.0: 23 destinations, 28 routes (23 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 100.100.0.0/16 Self I
* 150.150.0.0/16 Self 250 I
[edit] Remember this very important rule: OTC Local Role setting has an absolute priority over anything explicitly set in the BGP import/export routing policies. No matter how specific that setting is, OTC Local Role overrides this and acts as a safety pin for any wrongly constructed routing policy.
But as mentioned above, we have a solution — override OTC Local Role for the individual prefix 200.200.5.0/24 within the EBGP export policy towards AS1 and AS2:
root@AS100# set policy-options policy-statement PS-EBGP-TRANSIT-OUT term EXCEPTION then otc-local-role provider
[edit]
root@AS100# show | compare
[edit policy-options policy-statement PS-EBGP-TRANSIT-OUT term EXCEPTION then]
+ otc-local-role provider;
[edit]
root@AS100# show policy-options policy-statement PS-EBGP-TRANSIT-OUT
term STATIC {
from protocol static;
then accept;
}
term EXCEPTION {
from {
protocol bgp;
route-filter 200.200.5.0/24 exact;
}
then {
accept;
otc-local-role provider;
}
}
term BGP {
from {
protocol bgp;
community CUSTOMER;
}
then accept;
}
then reject;After committing, we get this:
root@AS100> show route advertising-protocol bgp 100.100.6.1
inet.0: 23 destinations, 27 routes (23 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 100.100.0.0/16 Self I
* 150.150.0.0/16 Self 250 I
* 200.200.5.0/24 Self 200 I
root@AS100> show route advertising-protocol bgp 100.100.7.1
inet.0: 23 destinations, 27 routes (23 active, 0 holddown, 0 hidden)
Prefix Nexthop MED Lclpref AS path
* 100.100.0.0/16 Self I
* 150.150.0.0/16 Self 250 I
* 200.200.5.0/24 Self 200 INow AS100 is nicely leaking the route 200.200.5.0/24 to its upstream transit providers AS1 and AS2, even though it is received from its peer AS200. And that’s precisely what we wanted to achieve.
Berislav Todorovic is a seasoned network architect, being active for more than 30 years in the industry. Currently working as a Professional Services Consultant at Juniper Networks, where he is actively involved in the design and implementation of large-scale IP/MPLS networks and complex service provider network migration projects.
Originally posted at LinkedIn.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.
I little bit understand