In many ways, running an ISP router is not much different from running a car. Both need to be continually maintained and secured, and the engineers or drivers need to be properly trained to allow for them to be used in an optimal way.
Below are some best common practices that I often share as an APNIC Community Trainer based on my experience managing ISP networks.
Secure your engine
For me, an ISP’s routers routing engine is equivalent to a car’s engine; they propel the packets through the network. As such we want to make sure they are well maintained and protected.
The most critical part to protect is the loopback IP address — it is the identifier and reference point for routing protocols to properly route the packet. Mechanisms to protect it are also known as ‘Control Plane Policing’ (Cisco), ‘Protect RE Firewall Filter’ (Juniper), or ‘CPU Defend Policy’ (Huawei), but the function is the same — to protect the control plane.
Remember, the loopback address needs to be protected both via IPv4 as well as IPv6 since the attack might be coming from both types of the IP address. Other things to consider when validating the packet from communicating with loopback 0 are blocking Telnet and limiting Secure Shell (SSH) from internal IP addresses, Internet Group Management Protocol (IGMP) and Simple Network Management Protocol (SNMP) queries, Open Shortest Path First (OSPF), Intermediate System to Intermediate System (IS-IS), Border Gateway Protocol (BGP) and other signalling protocols such as Bidirectional Forwarding Detection (BFD).
Like drivers, engineers should be qualified
A car is as only good as its driver. From time to time, consultants and architects will do their best to design and configure the router in the best possible way, but they won’t always be around to continually oversee this.
To drive a car legally, you need to get a driver’s license. And once you have a license there is normally a probation period. I would suggest that ISPs should employ a similar practice for the operators and engineers managing their network. One way to do this is to make sure they have been certified to operate and maintain the equipment.
Vendor certification is not an assessment of how good the engineer is, but is useful to ensure that the engineer is well versed with the equipment they are managing.
Next to this, experience is the next most important thing. To help build this experience, while protecting the network from any major outages, operators should control the access capabilities between vendors, junior engineers and senior engineers. This can be done by managing the client authorization manually inside the router or centralized via radius servers.
An example I often share to show the importance of operators applying these controls is an incident where a junior engineer had accidentally configured the WAN IP address from initially a /30 to a /3 where it had advertised 0.0.0.0/3 into the network. To give you an idea of the impact of this incident, imagine a car opening its door for all the passengers on a train to rush into.
You wouldn’t leave your car unlocked
Now that we have a well-secured engine and a qualified driver, it’s time to protect the passengers/cargo.
A car has a key or remote to ensure that only the owner is able to open its doors to whomever they desire. The same function needs to be applied for our routers to ensure no bad actors can come in or connect with us.
From my experience I’d advise, at the very least, implementing MD5 authentication for all interior gateway protocol (IGP), BGP as well as RSVP sessions connecting to your infrastructure. This is to ensure that no unknown customer-premises equipment (CPE) can plug into and get your router’s database or packets.
Enabling this line-by-line for all peering connections may seem like a hassle but if you consider the benefits gained from it, it’s well worth the effort. For Juniper routers, authentication can be applied via a group command and the rule can be applied within the protocol stanza thus minimizing changes as well as changing the key later. The owner, though, needs to ensure that nobody without the right key can come into the car and try to go into it without alarming the authority, which in this case, is our Network Operations Centre (NOC).
When was the last time your network was serviced?
We have covered the engine, driver, and the passengers/cargo. Now, we need to consider servicing our car.
Every new car has a service schedule that allows the manufacturer/mechanic to check and ensure that all the critical parts are working properly and, if not, are replaced or updated. The same goes for our routers, which requires having constant dialogue with your vendor or principal to ensure the router operating system is running the latest version and being patched properly for any vulnerabilities.
It’s not uncommon for certain car manufacturers to recall cars due to some defective parts. Likewise, vendors will do this, predominantly in software updates, but it still again requires constant dialogue. This includes sharing issues and workarounds with the vendors and the wider community. After all, we’re all connected together.
APNIC meetings offer a great platform for communities to meet up, share their experiences and learn about latest technologies and best practices from all the biggest providers and principals. Therefore, it is important for us to work together in securing our router and network, as well as the Internet.
Physical inspection is also critical, even those ratty paper labels that you’ve got wrapped around your cables. Try identifying a single fibre from a 48-core patch panel at 04:00 in the morning on a Saturday when all the cables have not been tagged or tagged incorrectly. There have been incidents where a primary link was down, and during the on-site troubleshooting, the vendor accidentally pulled out the backup fibre link due to it being labelled incorrectly. Have you seen the movie Titanic where the ship is sinking and suddenly it broke in half? That’s how the NOC felt that time.
Take a holistic approach to networking
The above is only the tip of the iceberg when it comes to maintaining a healthy and optimally running ISP — there are so many best practices we need to follow to secure our networks.
What I want to make clear in this post is that it is never a single part that requires monitoring, securing or fixing but is a group of related parts and events that need to be checked and maintained as a whole.
In the case of a car, it’s about making sure the car is well made, serviced, looked after and secured, as well as how capable the person driving it is (we’ll leave road rules for another post), all of which can relate to similar scenarios with running an ISP.
Muzamer Mohd Azalan is a network engineer with 8 years’ experience managing the core network for one of the biggest ISPs in Malaysia.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.
good writing and information. thanks!
Excellent comparison! Thanks for sharing it.
Very good article and comparison