If you are a network engineer working for a transit or service provider, part of your life is to make company business come to life by translating business rules into routing policy agreements.
However, business agreements are sometimes complicated, and the routing policy validation process is not formalized, time-consuming, and error-prone. Policy validation normally involves inspecting several looking glasses around the world and sometimes asking friends or clients to send you a traceroute to check the correctness of your implementation. Furthermore, routing asymmetry is always an issue lurking around the corner. And things get worse if you are in a region not well covered by RIPE Atlas or the NLNOG ring, such as Africa, South America, and Asia.
To improve and help in this validation process, we at the University of Twente, in collaboration with SIDN and NLNETLAB, built TANGLED — a cooperative anycast testbed able to provide you with a vision of 90% of all active Autonomous Systems (ASes) on the Internet.
Using anycast to validate policy routing
Whoa… isn’t anycast just a bunch of servers on the Internet using the same IP address? How can this help in routing policy validation?
Well, like any other service, anycast relies on BGP routing to forward users’ traffic to the anycast sites. Typically users are routed to the topologically nearest anycasted copy. But, if you apply a policy change to routing preferences, the chosen anycast site changes too, so we can map the effect of each routing policy on the Internet.
Now imagine being able to receive a view from 6M networks and more than 60k ASes towards your client or service. It’s a good way to test your routing policy, isn’t it?
Uhmm, ok… and how does it work?
The TANGLED testbed has its own AS and IP space and several sites spread over the world. When you set up a TANGLED anycast site as an AS-client of your network and apply to it the policy you want to test, you can learn which networks prefer the routing information you announced.
Our testbed consists of various copies (also known as anycast instances or anycast sites) distributed around the globe and co-located under different ASes, as well as a set of tools to:
- Provide a programmable anycast traffic engineering interface, able to control each individual anycast site’s visibility.
- Map the distribution of traffic from clients to the anycast sites using millions of vantage points.
- Validate the BGP traffic engineering configuration on our transit providers.
- Measure and analyse result data from experiments.
Figure 1 shows an example of its use:
Imagine you want to validate a routing tag that affects just clients, some private peerings, and a local Content Distribution Network (CDN) node. You just need to configure TANGLED to announce a more specific prefix on your local node and mark this route inside your network. All clients, peers, and the CDN are expected to receive and use this path, answering back to TANGLED Site-A, because this announcement is more specific. If you receive more or fewer answers than expected, something needs to be fixed. Easy, right?
Enterprises can benefit too
Enterprises often have some contracts activated on demand, for example, a backup transit agreement just used in emergencies such as a huge DDoS. Sometimes we discover these backup contracts are not fully operational right when we need it. Maintenance happens every day, and an unused path is an unverified path.
With TANGLED it’s possible to automate this validation and compare the historical data over time. Recently we applied TANGLED to better understand outages on Internet exchanges, but this is another story.
In TANGLED, we support several methods of BGP engineering customizable by each individual anycast instance. For example, AS-path prepending, route poisoning, reverse prepending, selective announcements, and any BGP community strings as supported by the IXPs or our partners around the world. Testing policies is a breeze thanks to our command-line tool that exercises centralized control over TANGLED — you don’t lose time changing individual anycast routing configurations.
Which type of data can I get from TANGLED?
We have also developed tools to explore the data generated from the measurements phase. All data is exported in comma-separated value (CSV) and its basic format is simple, containing pieces of information such as RTT, TTL, ipgeo-data, origin and destination site:
To demonstrate how easy it is to work with this data, we made Python Notebook examples available on Github:
We are constantly looking for opportunities to expand our testbed by establishing new partnerships, as well as deploying new nodes. Learn more about the testbed via our website and recent IEEE IM paper and leave us a comment below if you have any questions or recommendations, or wish to share your experience with TANGLED.
Leandro Bertholdo has over 20 years of experience in network management and operation, being responsible for long-distance networks, metropolitan networks, and IXP operations.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.