As an admirer of the Unix philosophy, I’ve always marvelled at the simplicity of its shell design — basic tools cooperating to achieve complex results.
Most network operators and researchers might share this kind of fascination in their everyday work but may also feel the lack of such well-behaving tools capable of performing trivial steps to a bigger combined result.
The network community has no shortage of amazing collaborative efforts that provide endless amount of data useful for analysis.
Specifically, Multi-threaded Routing Toolkit (MRT) dumps are a standardized format for compressed files archiving precious information on Border Gateway Protocol (BGP) routing tables at a point in time, and their subsequent evolution, as observed by route collectors.
Several route-collecting projects publish their MRT dumps freely, including RouteViews at the University of Oregon and the RIPE NCC Routing Information Service (RIS). These projects are a crucial source of data to assess network health and investigate routing security.
Whenever you run into an Internet reachability issue, BGP looking glasses are the usual place you go to start troubleshooting. Since we cannot travel back in time to identify the problem, the use of historical data and the subsequent flow of events, such as BGP updates, can help inform whether an incident is caused by a mis-origination or a missing route. Thanks to projects such as RouteViews and RIPE RIS you can have access to daily BGP snapshots from which you can extract routing entries to your network.
Despite the abundance of this well-understood data, most of the time, tools to operate on it are fairly complex pieces of software — possibly even network services with a strict set of features, where the only means for a user to customize the analysis process requires interfacing with a client API.
That requires:
- A basic knowledge of scripting in a specific language.
- Knowledge of the specific server web API exposed by the service.
In short, a whole dedicated knowledge base for the sake of a single tool. Shouldn’t it be possible to reuse all our hard-earned knowledge in the Unix shell and our playful hacking skills (to conjure scripts quickly), that work on publicly available MRT dumps and produce useful data sets for our purposes the way we want?
Wouldn’t it be nice to bring these useful new tools to our familiar grounds in awk, sed, grep, perl or python?
This is the gap Micro BGP Suite (µbgpsuite for short) attempts to fill.
The µbgpsuite project comes with three objectives:
- Performance. Good tools should not come with a prohibitive requirement set but produce interesting results quickly even on modest machines.
- Provide powerful analysis and research tools for BGP to make previously challenging tasks plausible, without alienating the user.
- Provide an extensive library to developers. Good tools should be extensible in the face of unanticipated needs.
And, of course, be free and open source, because we like to ponder on the implementation of our tools and customize them to our needs.
I won’t elaborate on the developer library here, as it’s targeted mostly at developers with specific needs. Instead, I will concentrate on its tools for operators and analysts:
bgpgrep— the star of the show.peerindex— an inspection tool for MRT Routing Information Base (RIB) dumps, to print out relevant entries of the Peer Index Table.bgpscanner— a backwards compatibility script to Isolario bgpscanner, for users familiar with the old bgpscanner utility by the same author. Its purpose is to ease the migration process and learning of its replacement tools.
Let’s focus briefly on bgpgrep, by far the most flexible among the tools mentioned.
Like the excellent bgpdump by RIPE NCC, bgpgrep is a friendly shell utility to format MRT dumps in a human readable way.
It inherits the same output format as bgpscanner, carefully defined to be parsed easily by scripting languages (like Perl or Python) or other shell utilities, making it a nice Unix citizen that can easily interoperate with the rest of the system.
Other defining traits are its performance (about 20% faster than the already outstanding bgpscanner), and its extremely flexible BGP filtering and matching capabilities.
For simple uses, bgpgrep can directly decode and format several MRT dumps efficiently, in such a way that output can be conveniently fed to more utilities.
In more advanced uses, bgpgrep can extrapolate only data of interest from a vast set of dumps, by applying rules defined with an intuitive syntax.
Extraneous data is discarded from output with minimal processing, making the use of rules a convenient time saver. Using rules can save between minutes and hours when several gigabytes of input are involved. Data can even be piped directly to bgpgrep; there is no actual need for it to come in an MRT dump file.
Examples:
- Print any BGP message with supernets of 8.8.8.8 originated after 10 July:
$ bgpgrep ./rib.20210701.0000.gz ./updates.*.gz -supernet 8.8.8.8/32 -timestamp ">=2021-07-10"- Print any BGP message whose AS_PATH contains loops or bogon ASN:
$ bgpgrep ./rib.20210701.0000.gz -loops -or -bogon-asn- Print messages with subnets of 2001:67c:1b08::/48 not originated by AS2598 (trivial mis-origin/hijack detection):
$ bgpgrep ./rib.20210701.0000.gz ./updates.*.gz -subnet 2001:67c:1b08::/48 -not -aspath "2598$"- Print messages whose AS_PATH crosses link AS174 AS3356 AS2914, with possible prepending in between (useful for leak detection):
$ bgpgrep ./rib.20210701.0000.gz ./updates.*.gz -aspath "174 3356+ 2914"Where to find them
If you are looking for some good tools to inspect MRT dumps, which integrate with familiar utilities the way you want, check out the Micro BGP Suite at DoubleFourteen Code Forge and have a look at its sources at the official DoubleFourteen Code Forge Git Repository.
I hope this post can be at least as useful to you as it was fun for me to write.
Adapted from original post which was first published on the MANRS Blog.Lorenzo Cogotti is founder and lead of the DoubleFourteen Code Forge, a non-profit community promoting independent research and good software practices.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.