Revisiting IXP peering LAN security in the light of new threats and technology evolution

Daniel Kopp and Matthias Wichtlhuber co-authored this post.

Internet Exchange Points (IXPs) and network operators want to make peering LANs as secure as possible by only forwarding desired traffic. Significant work was done over a decade ago when peering LAN security and port hygiene were hot topics in the IXP and peering community. During that time, Best Current Practices (BCPs), RFCs, IXP service definitions, configuration guides, and other community standards were developed. Since then, these documents have seen few updates regarding lower layers of network security, while the threat landscape and router technology have evolved in large steps, for example, with IPv6 adoption and EVPN.

This article revisits the assumptions on which peering LAN recommendations were based and identifies potential for improvement by analysing several years of Access Control Lists (ACLs) statistics at a large IXP.

Why is peering LAN security important?

A good peering LAN security design must mitigate the three major risks to IXP operators and their members, touching on both security and resiliency aspects.

Overwhelming member routers with control plane noise: Control plane noise is traffic that the router must process and then drop, either in hardware or more commonly in software. One primary example is Address Resolution Protocol (ARP), which can become problematic in large broadcast domains like peering LANs, because it is flooded through the network. If done wrong, especially for Layer 2 flooded protocols, the stability of the IXP’s routers can be affected too, in addition to members’ routers. Noise exists at Layer 2 and Layer 3, and the existing recommendations separate the treatment of traffic along these layers.
Protecting member’s routing: While it is common to apply route policies to BGP peers since they interconnect different untrusted networks, Interior Gateway Protocols (IGPs) typically have less filtering as they run between routers within the same network. This may lead to highly undesirable situations, for example, forming Open Shortest Path First (OSPF) adjacencies over a peering LAN. Similar risks exist for many other routing and discovery protocols. While IXP operators usually have a quarantine phase for IXP members to detect these issues, this is not enough as router configurations change frequently and can introduce unintended configuration changes.
BGP security at route servers: Route servers maintain several BGP sessions directly and have an exponentially growing number of paths for networks. Route server security is not only a relevant topic for the directly connected members but also for their upstreams and downstreams. The most common issues are route leaks and BGP hijacks. BGP security is currently where most innovation is happening, for example, BGP Prefix Origin Validation, Route Leak Prevention and Detection Using Roles in UPDATE and OPEN Messages, and Autonomous System Provider Authorization (ASPA). While route server security is an important topic, we do not cover it in this article and focus on security issues relevant to the local peering LAN.

To check how up-to-date the current recommendations are, we examined traffic patterns at a large IXP and conclude the article with proposals to change them.

Analyzing peering LAN noise

Figure 1 — Investigating traffic patterns via DE-CIX Frankfurt’s peering LAN.

To investigate traffic patterns, we observed intentionally unfiltered Layer 2 and Layer 3 traffic on a large IXP peering LAN, DE-CIX Frankfurt. There are currently over 1,160 connected networks providing an opportunity to observe a wide variety of routing implementations and router configurations. The measurements were done at two different routers with two different methods providing two different levels of detail as shown in Figure 1.

Both routers have the perspective of an IXP member and receive filtered Layer 2 and unfiltered Layer 3 peering LAN traffic. As mentioned, Layer 2 filtering is critical for the IXP’s own network stability, so removing these ACLs for experiments was out of the question. Thus, for Layer 2 we can only provide drop counter statistics from the IXP edge router but have no detailed insights on what was dropped.

Router A captured peering LAN noise over 10 days, while router B conducted long-term measurements with predefined ACLs at Layer 2 and Layer 3 for several relevant protocols over a several-year time frame. While the traces from router A give very detailed insights down to the packet level, router B only counted predefined protocol packets and the bursty nature of member incidents.

The plot on the right (Figure 1) illustrates measured OSPFv3 packets and highlighted several OSPF incidents where IXP members potentially misconfigured OSPF on their router. These incidents are bursty with long periods of no activity but also with incidents lasting a year or more. They are not visible in router A’s packet traces due to the limited time window and highlight the need for permanent monitoring of this type of traffic to alert IXP members.

Results

We analysed data plane and control plane traffic by looking at the packet counters for:

Frame types that should be filtered, like LACP, Layer 2 discovery protocols and MPLS.
Protocols and link-local packets that should be filtered, like Layer 3 discovery and routing protocols.

We investigated the captured data by starting with Layer 2 measurements. The BCPs/standards recommend only forwarding ARP, IPv4 and IPv6 EtherTypes (whitelisting) and only allowing the member’s router’s MAC address on egress as a destination MAC address and on ingress as a source MAC address.

The latter recommendation is critical since not allowing specific MAC addresses can result in Layer 2 loops if a member accidentally bridges two IXP ports. As there is no TTL counter on Layer 2 frames, they will circulate endlessly between the peering LAN and the member side. This is especially bad for broadcast protocols like ARP. In the best case, this only clogs the affected ports; in the worst case, it can lead to a complete outage of the affected peering LAN.

Our measurements show that only a very small amount of overall traffic falls into one of the categories to be filtered, however, this small amount is important to be kept away from IXP members.

IXP	Filter count total [share packets]	Total bytes [share bytes]
Ingress (member to IXP)	0.05%	0.017%
Egress (IXP to member)	0.000014%	0.0000015%

Table 1 — Filtering measurements on ingress and egress.

The important thing to note about Layer 2 filtering, is that it also provides adequate IPv4 Layer 3 control plane protocol filtering because most protocols have their own EtherTypes. Additional filtering could be applied to more obscure IPv4 routing protocols that use IPv4 frames. Proper filtering is more complicated for IPv6 because you need to filter on multicast addresses and/or the IPv6 Next Header for each protocol that is undesired. The common approach to filter recommendations on Layer 3 is to block all protocols that may leak from members’ internal networks into the IXP’s peering LANs. Which protocols to block varies widely, ranging from no recommendations at all to a few examples for specific protocols. No recommendation currently provides a complete list.

	Packet captures		Router ACLs
Protocol	IPv4	IPv6	IPv4	IPv6
OSPF	64.3%	–	32.38%	3.25%
PIM	2.56%	–	21.53%	1.43%
EGP	–	–	3.03%	–
IGRP	0.48%	–	18.1%	–
ICMPv6 MLD	–	–	–	84.73%
ICMPv6 RA & RS	–	–	–	10.59%

Table 2 — Protocol prevalence as seen in PCAPs and router ACLs.

When analysing the data, we find that OSPF is by far the most prevalent protocol to leak into peering LANs, followed by PIM, IGRP, and EGP. We find OSPF traffic in both the packet captures as well as the ACLs. New recommendations that are not mentioned anywhere are to also filter ICMPv6 MLD and ICMPv6 RA/RS.

Technology evolution: EVPN

Several IXPs have introduced EVPN over the last few years with great success. This significant development within the area of Layer 2 technology is currently missing within recommendations. One important aspect of EVPN is the handling and answering of IPv4 ARP and IPv6 ND packets in a central way in Provider Edges (PEs) at IXPs with static IP/MAC configurations. This avoids flooding these protocols and highly reduces noise in peering LANs, while also making neighbor discovery more secure. Figure 2 shows ARP traffic in the DE-CIX peering LAN since mid-2015. The big drop in traffic on the right is the introduction of EVPN in the peering LAN, reducing the ARP to a minimum.

Figure 2 — ARP traffic on the DE-CIX Frankfurt peering LAN from mid-2015 to mid-2023.

Conclusion

Generally speaking, peering LAN security recommendations still fulfil their purpose. However, there is room for improvement. We propose the following changes:

Recommendations for filtering Layer 3 protocols should be made explicit by mentioning a comprehensive list of protocols to filter and should provide examples for ACLs. Where IPv4 Layer 3 protocols can be filtered by Layer 2 EtherTypes, there should be Layer 2 recommendations as well to support older or restricted hardware.
Unwanted traffic should be monitored permanently. As shown above, initial quarantine is not enough since router configurations change frequently and incidents may go unnoticed for a long time.
All recommendations should take IPv6 into account, especially ICMPv6 MLD and ICMPv6 RA/RS and provide examples for filtering.
A widely available Layer 2 technology like EVPN should be recommended.

Though we specifically looked at peering LAN security, the same recommendations also apply to any interconnection links (public, private, transit, and so on). Developing and providing an in-depth strategy where all parties apply strict ACLs will provide layers of protection to make interconnection more secure for everyone.

Let’s continue the discussion in the peering community to improve peering LAN security!