Everyone who operates a network understands the importance of thorough testing. This becomes even more important for vendors of network hardware and software.
Most network providers start testing with a variety of scripts, using well-known open-source tools such as iperf, scapy or hping3. PPPoE or IPoE (DHCP) subscribers are typically emulated using common Linux clients with custom scripts for orchestration. There are also multiple open implementations of routing protocols available, such as the FRRouting project or BIRD.
Test engineers need to be familiar with all of these tools and frameworks. Some tools are limited to a single implementation, but most network protocols allow for multiple different valid behaviours, which means engineers need to emulate a protocol behaving as it might from both vendors A and B, without wanting to buy hardware from all the relevant vendors.
This approach works well enough until you reach a certain level of scale and complexity. A Broadband Network Gateway (BNG) is typically the most complex element in an ISP network. The BNG combines provider edge functionalities and routing protocols, such as ISIS and BGP, with legacy access protocols such as PPPoE and L2TPv2. A BNG is the service creation point for residential Internet access.
This sounds straightforward enough, but if you ask 100 ISPs how they build their residential access networks, you will likely get 200 different answers!
Such a service creation point is built using multiple legacy protocols and often those protocols have been used for different purposes than they were originally intended. Protocols such as DHCPv6 are designed for Ethernet, but using them within a PPP session requires a different implementation. The PPP protocol itself was also not designed as the client-server protocol it is used for today. Such a device, with legacy protocols accessible by millions of untrusted clients, requires a lot of security hardening.
In some economies, residential Internet service features are mandated by the government. This often includes strict requirements regarding accounting accuracy, legal interception, and wholesale services, which also require some compliance testing.
It’s also common for customers to have the freedom to connect their CPE, which requires providers to deal with hundreds of different client implementations. Their implementations are all a bit different and sometimes even wrong, but if CPE is used by millions of customers and has been working on previous BNG platforms, it is hard to argue that they aren’t behaving correctly.
With all of these requirements to satisfy, operators are typically forced to buy expensive network test suites with support for all those protocols, and for some requirements, they even need dedicated test hardware, which comes with enormous costs. It is quite common for the test equipment to be more expensive than the actual devices under test.
Time for a new approach
Here at RtBrick, we started our testing with the amazing open-source project TRex, which is a powerful traffic generator, but it didn’t allow us to do a lot of control plane tests. For these, we used a commercial network test suite without dedicated test hardware.
Such commercial network test suites typically need licenses for every port in use, which limited our test activities to the number of licenses we had. It was also not possible for us to modify or extend protocols, and some features we needed are only supported on the testers’ hardware chassis.
It became clear to us that we had to decide between making a huge investment in test software and hardware or using our know-how to build our own network test suite. We decided to make it on our own, so the BNG Blaster project was born.
This project was already showing good results after a few weeks, so we decided to continue on this path. Today, almost all our tests are run on our BNG Blaster, with thousands of fully automated test runs performed every night.
After two years of continuous development, all our investment into BNG Blaster was fully covered by the savings we made on commercial network testing hardware and software. We can also do tests that are not possible with any other test suite.
It was also clear from day one that we would share this work with the networking community. We decided to release it under a liberal (BSD) license to give the community the possibility to use, modify and repackage it with the maximum amount of freedom. Such a freely-available test suite enables operators to report issues to network vendors with an attached BNG Blaster configuration to allow vendors to reproduce the issue under exactly the same conditions.
We initially focused on typical subscriber-faced testing (BNG and IPTV) and gradually evolved to routing protocols and traffic generator features.
The BNG Blaster was completely developed from scratch, because we wanted to test for maximum scaling with a small resource footprint, in a way that was simple to use and easy to integrate into any test automation infrastructure.
BNG Blaster features
The BNG Blaster can simulate massive PPPoE and IPoE (DHCP) sessions, which could be using IPTV services, L2TPv2 (LNS emulation), L2BSA, QoS, forwarding verification, lawful intercept, and convergence testing capabilities. The traffic generator supports millions of separate tracked flows. This allows verifying every forwarding state of a full feed Internet routing table or sending traffic to every QoS queue of a full-scale service edge router.
The BNG Blaster supports:
- Emulations of massive sessions with low CPU and memory footprint
- Every modern Linux, virtual machine, and containers
- All protocols implemented in user-space and optimized for performance
- Automation friendly APIs
- Access protocols:
- Emulating massive PPPoE and IPoE (DHCP) clients
- Emulating L2TPv2 LNS servers with different behaviours
- Emulating A10NSP interfaces for L2BSA testing
- Including multicast and IPTV test suite
- Verifying legal interception (LI) traffic
- Routing protocols:
- Emulating ISIS topologies with thousands of nodes
- Supporting ISIS segment routing
- To set up thousands of BGP sessions with millions of prefixes
- Verifying MPLS labels for millions of flows
- Traffic generators:
- Generating and tracking millions of traffic flows
- Verifying your QoS configuration
- Verifying all forwarding states
- Measuring convergence times and loss
Get involved
The BNG Blaster is now used by leading network operators, network hardware vendors, and network software vendors such as Deutsche Telekom AG with their innovative Access 4.0 project.
This project will be actively maintained and further evolved by RtBrick. We are fully committed to building a project for the community and take issue and enhancement requests seriously. We are looking forward to any kind of contributions, new features, bug fixes, or tools. Even contributions to the documentation are more than welcome.
If you are interested in the BNG Blaster, or simply looking to find out more about it, we recommend going through the examples in the quick start guide.
Our mission is to build better networks with open test suites.
Christian Giese is a Senior Software Engineer with a demonstrated history in the networking industry.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.