Yesterday (6 June 2019), Swiss data centre co-location company Safe Host (AS21217) leaked over 70,000 routes to China Telecom (AS4134) in Frankfurt, Germany, some for over two hours.
China Telecom then announced these routes onto the global Internet redirecting large amounts of Internet traffic destined for some of the largest European mobile networks through China Telecom’s network. Some of the most impacted European networks included Swisscom (AS3303) of Switzerland, KPN (AS1136) of the Netherlands, and Bouygues Telecom (AS5410) and Numericable-SFR (AS21502) of France.
Often routing incidents like this only last for a few minutes, but in this case, many of the leaked routes in this incident were in circulation for over two hours. In addition, numerous leaked routes were more-specifics of routed prefixes, suggesting the use of route optimizers or similar technology.
At 09:57 UTC, over 1,300 Dutch prefixes were announced in this leak. For 470 routes of KPN (AS1136), the leak took the form:
… 4134 21217 21217 21217 21217 21217 21217 13237 1136
If someone thought the prepending of AS21217 would keep these routes from leaking out, they were mistaken. Below is a visualization of the propagation profile of a prefix (46.145.0.0/16) announced by Dutch carrier KPN.
Beginning at 09:44 UTC, over 200 Swiss prefixes were announced by this leak. For 64 routes of Swisscom (AS3303), the leak took the form:
… 4134 21217 21217 21217 21217 21217 21217 6830 3303
Below is a visualization of the propagation profile of a prefix (46.14.128.0/17) announced by Swisscom.
Each of these plots depicts peaks and valleys in the spread of propagation over time. These ‘features’ are caused by changes in how China Telecom exported these routes to Tier-1 telecoms during the leak. Below we can see the preceding graphics side-by-side with equivalent views from the upstream perspective of AS4134.
Beginning at 10:10 UTC, 150 Bouygues Telecom (AS5410) prefixes were in this leak — 127 of which were more-specifics of existing routes. For example, 176.171.75.0/24 is not normally in the global routing table and is a more-specific of 176.128.0.0/10, announced by Bouygues Telecom (AS5410). These leaked routes took the form below:
… 4134 21217 21217 21217 21217 21217 21217 25091 5410
Users of these services began noticing problems and reported seeing traceroutes to their networks traversing China:
Bouygues Telecom:
Aujourd’hui de 12h20 à 13H30 les routes internet a destination de mon IP #Bouyguestelecom routé systématiquement par le réseau chinanet. @Bouyguestelecom avez-vous des informations sur une éventuelle annonce BGP suspecte pour la plage 128.78.0.0/15 ? #bgp @lafibreinfo @bgpstream pic.twitter.com/IUUh2E5B0a
— Cmer ⎈ 🐳 (@cm3r) June 6, 2019
KPN:
Aujourd’hui de 12h20 à 13H30 les routes internet a destination de mon IP #Bouyguestelecom routé systématiquement par le réseau chinanet. @Bouyguestelecom avez-vous des informations sur une éventuelle annonce BGP suspecte pour la plage 128.78.0.0/15 ? #bgp @lafibreinfo @bgpstream pic.twitter.com/IUUh2E5B0a
— Cmer ⎈ 🐳 (@cm3r) June 6, 2019
Mijn VPN tunnel vanaf #haarlem naar een klant met #KPN in Amsterdam gaat via #China? pic.twitter.com/tf6uhqThSM
— Wieger Bontekoe (@wbontekoe) June 6, 2019
Swisscom:
@Swisscom_B2B_en Are you having routing problems? We have a hard time reaching multiple servers. BGP Hijack ? https://t.co/0EcyQAzP8s ?
— Didier Raboud (@OdyX_) June 6, 2019
Many of our traceroute measurements were also sucked into this routing leak. The traceroute below begins in Google in Ashburn, USA, and is destined for Vienna, Austria, but is detoured through China Telecom (hops 5-8) en route to its destination.
Below is another traceroute from an Oracle data centre in Toronto to Numericable-SFR in France that gets diverted through China Telecom (hops 8-10).
BGP route leaks still remain a major problem
Today’s incident shows that the Internet has not yet eradicated the problem of BGP route leaks. It also reveals that China Telecom, a major International carrier, has still implemented neither the basic routing safeguards necessary both to prevent the propagation of routing leaks nor the processes and procedures necessary to detect and remediate them in a timely manner when they inevitably occur. Two hours is a long time for a routing leak of this magnitude to stay in circulation, degrading global communications.
A great place for any telecom to start improving their routing hygiene is to join the Internet Society’s Mutually Agreed Norms for Routing Security (MANRS) project.
Take the Introduction to MANRS course on the APNIC Academy
Adapted from original post which appeared on the Oracle Blog.
Doug Madory is a Director of Internet Analysis of Oracle’s Internet Intelligence Team where he works on Internet infrastructure analysis projects.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.