Network design: Dual ISP, DMZ, and the network edge — Part 1

Ready to live on the edge? In my last network design post we talked about remote access VPN but in this instalment, we will take a detailed look at designs for the network edge.

What is the network edge? The network edge is where your network and outside networks connect. In the enterprise world this is your path out to the Internet; in the provider world this is generally where you connect to upstream providers or peers. In this three-part post, we will cover high- and low-level designs, different types of topologies such as SMB, enterprise and Service Provider (SP), and look at the building blocks, redundancy options and other considerations.

I’ve seen several posts over the years on r/networking asking questions like what I do when I have dual ISPs, or how can I get redundancy with two Internet providers? How do I failover? I feel this is probably one of the less understood areas of networks for a lot of folk, perhaps because Border Gateway Protocol (BGP) is integral to it. Some might struggle with how to perform failover properly or there are just critical details that are often overlooked. I intend this series to be a detailed guide (as much as possible) to try and provide everything you need to understand and help design your network edge.

I will explain some things for those who aren’t familiar, but some topics are too deep to explain or cover in depth, so there are some assumptions regarding hardware sizing, cabling, routing/switching, and firewalls. This series is for educational purposes only.

Basic edge requirements:

1. Securely connect the internal network to other outside networks.
2. Provide a secure Demilitarized Zone Network (DMZ) delineation between internal and external networks.
3. Optional: Provide a secure path for outside networks to your edge services.
4. Optional: Provide redundancy and resiliency for Internet/peering.

Edge design challenges:

1. Routing with multiple providers or paths.
2. External to internal routing and internal to external routing.
3. Firewalls, asymmetric routing and Network Address Translation (NAT).
4. Connectivity options.
5. Budget.

Topics we will cover in this three-part series:

1. The foundation — ISP, diversity, IP addressing, redundancy and DMZ.
2. Branch high-level design.
3. SMB business high-level design.
4. Enterprise high-level design.
5. Service Provider high-level design.
6. Low-level edge examples and details.
7. Low-level DMZ examples and details.
8. Closing.

Laying the foundation

Before we get into some designs and options we should first talk about the foundation of the edge. Before you get to the first building block it’s always good to review the business goals, why do you need network edge, what goals does management have for the business or location — will it grow? Are there mission-critical applications or users this is supporting? How much downtime can be afforded? Getting some of those questions answered will help determine the requirements for the build.

Usually, you will have a firewall(s) (I hope!), possibly router(s) and various types of switches, depending on your goals and network size, of course. There is also the other side or other provider’s equipment — how will the circuit connect from them to you?

Circuits

That is the first building block, who is your provider? There are so many options these days but generally, if you’re an enterprise you need one to connect to the Internet, and if you’re a provider you have to connect to someone else to reach their network. What type of Service Level Agreement (SLA) will there be for the circuit? Or do you need mission-critical 99.99% uptime? That would be more costly than just a broadband-type SLA and that broadband circuit could take longer to repair in the case of an unexpected outage.

How will they hand it off to you — will it be a copper RJ45 connection or a fibre handoff? Consider if the Minimum Point of Entry (MPOE) into the building is 500ft away from your telco closet or your primary firewall; if so, you will need to extend from there into your data closet, however, copper will not go that far, therefore you would need a fibre hand off.

Where will you put the circuit? If you’re an SMB you can probably plug it right into your firewall, however, if you’re an enterprise or this is like an Internet Exchange Point (IXP), you will probably have it go into a switch in the DMZ or a border router.

In that case, what if another provider installs their demarcation equipment in that same MPOE room. Is there backup power in there? Because during a power outage if your main telco MDF has backup power but that equipment does not, then you probably won’t have a connection! So it’s important to consider the provider or peer’s equipment and placement during your initial design.

Usually, if the circuit is in a data centre (DC) it will have backup power from the data centre host and the same goes for cross-connects. Cross connects in the DC are usually single-mode fibre (SMF) and Layer 2 connections. Remember to consider the cross-connect monthly cost in your total cost if operating in a DC or similar.

Diversity

When you have multiple ISP connections coming into a location or for the same network it will be good to scrutinize how they are built on the provider’s side. You probably will want diversity between these connections so a failure isn’t shared between them. If both fibres are on the same aerial telephone pole and a car takes out the pole then both connections will be down. Sometimes it will be difficult to build into a single building from a different path, so the connections might come into the building in the same room but beyond that perhaps diversity can be built in.

You’ll want to get the local wiring service centres and Central office (CO) or point of presence (POP) where the circuits go and eventually terminate. You can work with your providers to ensure the redundant connections you have go to different equipment and buildings and hopefully can use different fibre beyond the ‘man hole’ or ‘street’ if two circuits are on a single building (that is usually a term for on the edge of the property).

Even with two locations in different geographies, it could be possible for different providers to have their circuits use the same POPs or DCs, so it’s important to have this built into your design when using multiple connections on the edge. In Figure 1, we can see a fibre map example with POPs shown; maps like these can help look at diversity.

When in DCs, it will be important to ensure your cross connects use different equipment as even with different Internet providers that could be a single point of failure. Work with your DC team or possibly IXP team (for some SPs) to help ensure proper redundancy. Speaking of DCs, they can sometimes offer an ‘IP blend’ service where they will facilitate the edge connection and then they peer with multiple providers on different fibres/devices for you. This can be a good option in certain scenarios for already built diversity.

IP addressing

The next building block is your edge IP address scheme. A good thing to do if you are a larger outfit or will have dual connections is to obtain provider-independent IP addressing from your Regional Internet Registry (RIR) like ARIN, APNIC, RIPE, and so on. Therefore, you can announce yourself via BGP to the world with IP addresses that your organization holds, however, this could be costly if trying to obtain a /24 or a /23 (or more) of IPv4 these days. As for IPv6 blocks, they should be handing those out like hotcakes.

In my opinion, it’s easier and gives you more options to get the failover you want with your own address space. You would also need to obtain your own BGP Autonomous System Number (ASN) as well if taking this path.

One important thing to do before deploying your own IP address space is to submit your prefix and ASN (via your Internet registrar ) into the Internet Routing Registry (IRR) database. This will indicate which ASN your IP space should be originating from. This is important for documentation and for providers’ policies on the Internet for accepting prefixes.

To take it a step further, especially for the SP operators, looking at Resource Public Key Infrastructure (RPKI) for your IP policies is an important part of the future of Internet routing architecture.

To continue, if you are a smaller enterprise or this is a smaller remote location then usually you can obtain a /29 or /30 address block for the site, however, in this scenario, you usually cannot use BGP and will have to use another failover mechanism for that.

Also, if building a DMZ, will you use public or private IP addressing? If you’re able to get /48 IPv6 blocks or /23 IPv4 blocks then you probably will have plenty of addresses and options for routing between multiple locations and running services on the edge, but if not then you will have to use private addressing, which would have some limitation if running NAT on the edge — you will see that in the high-level design section.

BGP and failover

Speaking of announcements, how will the outside networks or the Internet know how to reach you? BGP will be the standard way to do it. The first question for the builder is where will you announce your addresses from? If building a DMZ, will it be the firewall or the border router? If from your edge router, will the provider know the next hop inside the DMZ to reach your networks or NAT subnets? Or, if you are only announcing from your edge routers and performing NAT on your firewall, how will that work?

When using two provider connections you will usually prepend your ASN in BGP on the backup connection, so you don’t receive return traffic from the Internet on your secondary connection. This is because AS-PATH is used in the BGP route calculations and having a longer AS-PATH means the route will be less preferred by peers. You can also advertise shorter prefixes on the backup connection if you have the option as those will be less preferred. I like that because both routes are in the global table, so you have better resilience and faster pick-up for backup scenarios.

Then for outbound traffic, you will use the local preference from the provider-received routes in order to ensure your traffic leaves from the right connection. High local preference is preferred over lower local preference and is applied on your side. Weight often comes up but that is only for the configured router and not recommended. Both prepending and local preference are applied with a route-map, which we will touch on later.

Sometimes, though BGP is not an option, you will likely have to use static routing between you and the provider. The provider will then announce your assigned IP prefix to the Internet. The good thing about using BGP is you have several options and knobs for control. With static routing you will be more constrained and have fewer options when it comes to having multiple connections, but it can be accomplished.

In the provider network, it will be good to know if this connection needs preference or if you are receiving those routes from other peers. For example, you want to prefer certain prefixes from your new peering neighbor, however, for other parts of your network, those prefixes are still preferred via another transit neighbor. Also, what routes will you be announcing? Will you be aggregating or not? Could that mean your peer receives more preferred announcements elsewhere if aggregating? Often BGP communities will be applied in order to carry edge information throughout the network; this is something to consider if you aren’t doing that now.

For both scenarios, it will be important to see what the neighbor’s BGP policies are to help craft your configuration and design. We will talk about some of the BGP configurations in Part 3.

DMZ and edge design options

The last building block is really up to the designer and the various factors as discussed such as the requirements and what type of network is being built. A larger network with more connections and resiliency requirements will typically have a set of dedicated DMZ switches for the various router, firewall and ISP connections. Although, if budget is an issue, these switches can also be used in another capacity. For example, a service leaf in a spine/leaf design or a core switch in a tiered design, using an isolated VLAN for the edge functionality.

It’s important to size your hardware based on your current needs, but also for future expansion. You won’t want a router limited to a certain bandwidth level when you might need more two years later, or a firewall that can’t handle possible user session growth over the course of a new year celebration. Alternatively, if you are building for 1Gb now but might need 10Gb in the future, will the device line card support that? Always take a minute to think about future needs when building.

In the next part of this series, we will look at various high-level edge connection examples — from one connection to multiple connections.

Brandon Hitzel (Twitter) is a network engineer who has worked in multiple industries for a number of years. He holds multiple networking and security certifications and enjoys writing about networking, cyber defence, and other related topics on his blog.

This post is adapted from the original at Network Defense Blog.

Rate this article

---

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.