Interconnection agreements at scale: secret or simple?

By Martin Hannigan on 26 Oct 2017

Are settlement free interconnection agreements at scale, or otherwise, really a secret? They can be. This article provides a peep hole into the world of codified private interconnections. Did you learn something? Let us know.

By Martin Hannigan and Job Snijders

Contributors: Patrick W. Gilmore, Arnold Nipper, Christian Kaufmann

Settlement-Free Interconnection (SFI)(FCC, 49, 98) is a way of life for some Internet networkers. It is how the business of the Internet works. It is also a craft. Some view SFIs as a dark art (Lohdi, 1), where it is impossible to learn what happens behind closed doors. Who is in that smoky backroom (Lohdi, 1)? Network operators of all kinds. What is being discussed? Business. Is it difficult to learn? It can be. While it is true that the world of SFI and the agreements that make it happen are secretive (Brodkin, 1), it is also possible to peek behind the curtain. If you know how to look. Of course, a few pointers can be useful.

SFI is both a business and technical relationship between two networks. It is an agreement where the networks concede to trade traffic between each other’s customers without payment or without ‘settlement’. There are multiple steps to creating an SFI relationship. There are also many ways to walk down those steps. To keep things simple, we are going to look at only Private Network Interconnection (PNI). Other forms of SFI, for example, peering over a public Internet Exchange, are valid, but the majority of traffic on the Internet flows over PNIs. Without private connection based SFI, the Internet would be centralized across intermediate platforms. The risk would be higher. Performance could be lower. Costs would be substantial. By using the SFI approach, we fire on all cylinders. We reduce risk and cost widely.

SFI relationships require more than just knowledge of how BGP works. Business realities must be understood and mixed skillfully into the recipe. This includes technical aspects to ensure the proper outcome. Simply throwing fibre over a cage wall worked 25 years ago. Today, lawyers, accountants, project managers, and performance testers are stakeholders. They are part of the conversation, add collective value to the business decision, and, as a collective, they ensure that an agreement is as comprehensive and practical as possible.

Not all SFI agreements are created equal. Many are based on handshakes or email. Some are small memorandums that document non-binding terms. Others are intense, containing much legalese. The agreements can be complicated. Some are more prescriptive than you would expect. Every business has standards and they differ. It is not shocking or odd that the agreements are different. The more complex agreements seek to set a cadence of predictability, supporting business and network security, stability, and resiliency. The most useful agreements are fair and executable, based on the ideology of mutual benefit.

Objectives of an interconnection agreement

An SFI agreement defines the scope of interconnection activity. While it is impossible to think of every benefit, a broad tone can be set. The intent is important. It sends a message that the parties will act mutually, even in the event of silence on a matter that may arise later. Networks using SFI agreements need to: spend time thinking about what they want and why, understand how an agreement affects the future, and the associated benefits.

Objectives for an interconnection agreement to consider:

Provides for cost savings and performance improvements
Ensures the exchange of traffic is secure, stable, and resilient
Establishes timely cooperation for security and network incidents
Grants network traffic control rights
Usually, includes a non-disclosure agreement
Embraces The Law of Least Astonishment

The Law of Least Astonishment (Geoffrey, 1) should be a goal of any agreement. An uneventful SFI arrangement is always best, although it is not always possible. While you may find that you can create a long list of needs, it is efficient to mimic what has succeeded in the past. Smaller can be better. Generalities can be useful to capture scope. Cooperation on security and routing events makes the parties stronger. Enabling capital management and planning is a priority. In reality, our world of Internet networks is anything but predictable. It is important to achieve a level of consistency where possible. The SFI approach does its best to support this.

Business terms are needed as part of the objective and are negotiated by the network owner, normally the team of engineers and others responsible for the health and welfare of the network. They negotiate utilization, capacity, and management parameters, while legal terms are negotiated by lawyers. While plain language is always best (Bradner, 1), legal language is what makes it an enforceable agreement. Networks should cover all the necessary business, technical and legal points in the scope, such as term, jurisdiction, and venue. It’s important to include all necessary parties in the conversation. ncluding a wide audience in the conversation helps to set realistic business goals. Educating people all along the process also makes it easier to get buy-in.

General requirements

Having a list of prepared requirements is the first step to begin a conversation. It sets the expectation and intent, and describes the methods and defaults of how you will interconnect. There are no surprises here. The requirements are common and well understood. Start with the requirements that are written down. From there they can be tweaked and agreed to in writing.

Requirements included in many SFI agreements:

Peers must operate an Internet Protocol network and use Border Gateway Protocol version 4 (BGP-4 as defined in RFC4271 and updates) to manage the exchange of traffic at the interconnection points for all Internet traffic.
Peers must operate a managed 24×7 NOC and must remedy reported issues within a reasonable timeframe.
Peers must use mutually agreed Autonomous System Numbers (ASNs) at each interconnection point and announce a set of routes that should be consistent with publication in an IRR.
Peers must announce all routes, including customer downstream routes.
Peers must apply routing filters facing their customers to ensure correct announcements from its customers.
Peers must agree not to abuse the interconnection.
Activities such as pointing a default route at the peer or otherwise forwarding traffic for destinations not explicitly advertised, resetting next-hop, selling or giving next-hop to anyone is forbidden.
The scope of an interconnection arrangement can be global or localized to a specific region.
Public peering interconnection by Internet Exchange Points should be converted to the PNI once the parties reach 15% or higher average utilization for about three months.
Peers should make upgrades at the 95th percentile utilization rate of 65%, measured over a rolling three-month period.
Parties should provide each other with access to a BGP looking glass to enable debugging operations.
Peers agree to perform capacity reviews at least once a year, if not more, as needed.
Each network absorbs their costs.
Cross-connect fees are equitably split on PNIs.
Maximum latency to the next-hop should be no more than 5 (or X) milliseconds RTT. If the peer shows higher roundtrips, the peer might be engaging in remote peering, which raises a red flag: “Is this the right place to interconnect?” “Why are you transporting this so far?” Exceptions can be made. Transparency matters.
Peers should agree on a common automation source for all add/move/drop peering, for example, PeeringDB.

None of the requirements above is set in stone. They are business terms you would seek to negotiate. Some networks have higher or lower defaults. It can take time for a network to do an upgrade. Parties will tweak parameters to address provisioning, capital planning, and other needs. Security may be more important and you’ll set an expected response time for a priority issue. You might set the PNI utilization rate to 70%, but lower the measurement that triggers an expansion PNI. There could be language about new traffic and immediate demand. One might argue that for sudden needs to be accommodated, the other party might trade something else of value. The SFI dance is a tango of two networks. While it is not always beautiful, it usually works.

Common blockers

It is likely that you will encounter blockers in conversations with a few networks. Some networks publicly post the conditions of interconnection agreements (Comcast, 1). Some interconnection agreements state that if one meets the conditions, they still may not interconnect (Comcast, 3). Others do not publish their requirements (Cogent, 1). Many networks continue to rely on conditions that were useful a decade ago and in a way that reflected the traffic patterns of the day. While these conditions may continue to be relevant, they can sometimes be more onerous as a result of being less than important. All networks are different. While some may have good reason to insist on a condition, others will be more pliable and flexible. Your mileage is going to vary. Whatever approach you take is squarely up to you. If you choose to object to a condition that may be a blocker, research it. Gain confidence. When you make your opinion known, on either side of the table, have confidence in your belief. It should only serve to convince a party that you know what you are talking about.

Examples of common blockers used to deny interconnection requests:

Downstream ASN demands
Some interconnection requirements include the demand for a (large) number of downstream ASNs. If there is a mutual benefit to the interconnection, the number of downstream ASNs is not relevant technically speaking. A peer could deem it relevant for some other reason. Its relationship to traffic is null.

Backbone requirements
Some networks use the Internet instead of a physical backbone. While a backbone and the resulting network control has benefits, what matters most are the benefits of the traffic exchanged. If a party can control their own traffic by localizing it and demonstrate mutual benefits, what is the harm?

Lack of balance (ratios)
Some peering agreements include requirements that the amount of traffic exchanged in each direction be balanced. For instance, traffic in one direction cannot be more than double the traffic in the other direction.This requirement was originally justified because hot-potato routing and centralized web servers caused unbalanced traffic flows. In today’s Internet of distributed content, CDNs, edge caching, and other techniques, the requirement for balanced ratios (Streenbergen, 29) may not be applicable.

At the end of the day, SFI is a business tool to further the goals of the corporation(s). If a network can increase revenue or decrease costs by turning up an SFI link, staff would be doing their company a disservice by not agreeing to interconnect. If there are performance, income, or expense benefits to be gained, it must be taken seriously in the interest of stakeholders and shareholders. However, when the business case fails, it is fair to adjust the parameters. Fairness is key, as any large inequities will later show themselves and degrade the benefits over time.

Workarounds to interconnection agreements

Refusing to interconnect can be cultural. There may be a fear of the regulator. If after all these years of fighting interconnection, a network suddenly changes its it can create concerns. It can peak interest by shining a spotlight on current practices. If other customers hear of another getting an SFI relationship, it could create a ‘run on the bank’, sparking chaos for multiple relationships. It may also impact their value. External misunderstanding of network relationships has a wide variety of impacts. During periods of consolidation in the industry, interconnection agreements can impact value or affect an acquiring party in unexpected ways, such as acquiring a settlement-free relationship of a customer. While getting around these concerns is trivial, you don’t get the same security as you would with a formal agreement.

Existing service agreements can mimic peering and interconnection agreements. Parties can add, drop, or change terms to an MSA on renewal. Who cares what it is called as long as the same benefits are provided, right?

A party can add an amendment or other equally innovative language that results in benefits being provided. It enables the parties to interconnect without a formal interconnection agreement. It documents the interconnection and becomes the agreement. It may not be as detailed as we would like, but it serves as the same assurance. It is a baby step in the right direction if a formal declaration is desired in the future.

Examples of negotiable language that can be added to a services order:

Upgrades required at 65%
Parties must meet regularly to review interconnection
24 x 7 x 365 direct NOC access required
All routes including self and downstreams
Language about on-net vs off-net billing characteristics (ports, traffic, bits, access)
Must include all reasonably expected routes

When to use an interconnection agreement

Interconnection agreements aren’t for everyone or all situations. They can create unnecessary legal overheads. Handshakes work when you are not overly concerned with the loss of an ASN’s traffic via a direct PNI. Peering at exchange points is usually done by a handshake and should remain that way. Larger, private interconnection is different. Networks which have traffic, that if lost, would dramatically increase costs or substantially reduce performance should look to formalizing the relationship with an agreement. Keeping overheads low is important, which is why the scale is applicable when determining the need for a formal agreement. When importance outweighs overheads you know it is time to formalize.

Examples of when to use an agreement:

When you have large volumes of traffic significant to a network’s business results
To protect capital and operational expenses and investments in the relationship
When seeking to assure traffic and growth for business planning
To ensure access protection in the event of an M&A
To establish terms of mutual benefit(s) and what they are
To maintain performance
When a relevant party asks you to

How hard is this going to be?

Asking for private interconnection can make or break your credibility and the network relationship. It is important to engage and convince all parties of the idea. It takes an effort to start and finish an agreement. While it might be tempting to blurt a request at the other side, it is not always helpful. Avoid blindly sending email to a peering@ email alias. The shock and awe factor never helps. It also creates a record. The other party may be amenable to finding another way to achieve a common goal without calling it peering. Putting it in writing before discussing makes it a more cumbersome and formal process. Organizations need time to think about and discuss requests with their colleagues. Many questions need to be weighed. Creating a conversation rather than a demand is going to be the first step towards success.

Agreement terms

Agreement terms are variable. Ask for a term (length of agreement) that is reasonable. There are public references setting baselines: Google and Level(3) signed a multiyear^b agreement (Level3)^a. The Federal Communications Commission opened Charter’s network for seven years (FCC, 63, 132). The Charter files all interconnection agreements (FCC, 65, b.135). Read the filings and see how you compare or what you may be able to glean for your own baselines.

Making interconnection decisions

Know where you need capacity and where you don’t. In the case where capacity no longer makes sense, be sure to consider falling back to another connection. In all cases, use sound business judgement. Pushing and pulling the levers of an interconnection agreement should help you to operate your network more efficiently. Before making rash utilization judgements, understand the value of an underutilized link. Will that 1 Gb/s of traffic now be hauled over a backbone? Are there premium services contained in that link that will suffer performance wise and trigger an SLA with a customer? You may consider some rules of thumb when making such decisions.

Traffic risk reference table

Set up your own reference table. Establish how much used and available capacity you have. Once you figure that out, create step levels of risk using spaced percentage points, for example, 1%, 5%, 10%, 20% and so on.

IMPACTED USED CAPACITY	TOTAL AVAILABLE CAPACITY	PCT TRAFFIC IMPACT
1 Gb/s	100 Gb/s	1%
100 Gb/s	1Tb/s	10%
500 Gb/s	1 Tb/s	50%

Table 1: Traffic risk reference table

Traffic cost reference table (cost assumed at USD$2.00 per MB/s, effective cost)

Once you build your own reference table, also map costs to it. You can scale your impacted capacity against your average cost and come up with a financial impact. Use both to understand risks and costs related to network decision making. This should also include potential income.

IMPACTED CAPACITY	PAID TRAFFIC IMPACT
1 Gb/s	$2K MRC / $24K YRC
100 Gb/s	$200K MRC / $2.4M YRC
500 Gb/s	$1M MRC / $12M YRC

Table 2: Traffic cost reference table (Monthly and Yearly Recurring Cost)

Summary

The idea that all interconnection is a wink and a handshake is a myth. While that approach may work for setting up and tearing down BGP sessions on an IXP, it does not work to protect critical business interests. Operating a network at scale is hard. Removing as much risk as possible is key. Using balanced, well thought out interconnection agreements to reduce that risk should be a priority.

Other considerations

Good relationships play a part in interconnection. Cultivate and maintain them; they are an asset. You will use them time and time again. They will become key to your career. It is true that the squeaky wheel gets the grease, but that only works once. Be a good partner, behave like a respected network enabler. You will get extended mileage from your interconnection reputation time and time again.
The Internet has changed and keeps changing. So has the relevancy of the methods and procedures of the past. Denser peering and interconnection is the answer to less backhaul, and more billable traffic to downstream or less expense is one of many underscores to mutual benefit. Admittedly, some will never feel that mutual benefit is established without parity measured by a list of check-boxes. When there is mutual benefit, not much else matters other than exploiting that to the fullest potential. In many networks, interconnection remains a culture from 2001. Education will help somewhat. Pointing to the change in traffic movement including rapid Internet bypass uptake may awaken others.
The days of describing networks by ‘tier’ are over. It was an indicator of stature for network operators. The traffic patterns of the last ten years have dramatically shifted. It is true that technical IP networking is still the same, but nothing else is. The rise of the cloud and CDNs as fast, best performing, bypass or on and off ramps to the global Internet have impacted flows. In spite of a widely held belief that the middle layer is king, this is no longer the case for most. Cloud and CDN are successfully bypassing many directly to the eyeballs including SFI with the enterprise. They are directly connecting through settlement-free peering regardless of stature.
Ask peers to abide by the Routing Manifesto http://www.routingmanifesto.org/manrs/. We have too many security incidents that could be prevented. MANRS seeks to improve operations and reduce incidents. Read it, read it again, and try to codify its relevancy in your agreement.
Conduct regular reviews of peering policies. The peering policy forms the basis on which the agreements are based. Reviews are recommended to ensure the policy and the business objectives still align. It also helps to maintain a regular cadence of mapping policy to changes in the Internet, the traffic flows, the law, and culture. Do not be afraid to push back. As long as you have the proper data points and the context. Understand how they apply to the relationship. You won’t cause damage.
Coordinate with your internal teams. Check with your accounts and sales teams to make sure you understand the relationship with a peer. What crossover spending do you have? Are you asking for a benefit that is worth $1 and the other party buys $100 worth of your product? It is prudent and savvy to be sensitive to the balance of trade. When the sales team gets rankled, you’ll hear about it. Your partner will also use it as a technique to avoid interconnecting or to push fees in your direction. Carefully review the balance of trade and your strategy.
Challenge and respond. If you have strong arguments, you will have confidence in them. Some parties may try and weaken your arguments by denying them. That can be effective as it can create a logjam that invites doubt. Have confidence that you are correct. Be careful not to confuse confidence with arrogance. Be openminded to being wrong, but keep your eye on the ball at all times. You aren’t likely to need to surrender a point entirely unless it makes sense. You may need to propose modifications as a sound counterargument.

The Internet, interconnection, and peering is a business. Act accordingly.

References

Level 3 “Level 3 and Google Reach Settlement-Free Interconnection Agreement”, Jan 15, 2016. Accessed October 18th, 2017 (link no longer available).
Federal Communications Commission “FCC 16-59”, May 10, 2016. Accessed August 22, 2017 http://www.citationmachine.net/bibliographies/236587290?new=true
Comcast “Comcast Peering Policy” October 2013. Accessed August 23, 2017. https://www.xfinity.com/peering
Charter “Charter Peering Policy”, Unknown, Accessed August 24, 2017 (link no longer available).
Cogent “Cogent Peering Policy”, Unknown, Accessed August 24, 2017 (link no longer available).
Brander, S. “BCP 14”, March 1997, Accessed August 29, 2017. https://www.ietf.org/rfc/rfc2119.txt
Brodkin, J. “Why YouTube Buffers”, July 28, 2013. Accessed October 3, 2017. https://arstechnica.com/information-technology/2013/07/why-youtube-buffers-the-secret-deals-that-make-and-break-online-video/2/
Geoffrey, J. “The Tao of Programming”, 1987. Retrieved October 12, 2017
Lodhi, A., Laoutaris, N., Dhamdhere, A., Dovrolis, C. “Complexities in Internet Peering: Understanding the “Black” in the “Black Art””, In Proceedings of IEEE Infocom 2015. Accessed October 10, 2017. https://www.caida.org/~amogh/papers/peeringcomplexity-INFOCOM15.pdf
Steenbergen, R. “The Folly of Peering Ratios”. Accessed October 16, 2017. https://www.slideshare.net/RichardSteenbergen/a-guide-to-peering-on-the-internet

Editor’s note (18-Oct-2017)

a) A suggestion was made to cite a first hand source. A citation reference to lightreading.com was replaced with a reference to a press release from Level 3, detailing Level 3’s interconnection agreement with Google.

b) Initially a ten year contract term was mentioned in this article. While there is one public source suggesting a 10-year term, there is no second source to support the assertion. The phrase ‘ten-year’ was replaced with ‘multiyear’ to align with information made available to investors.

Acknowledgements

The Internet operates best when there is cooperation and collaboration. We thank Patrick W. Gilmore, Arnold Nipper and Christian Kaufmann for their time and efforts. They provided feedback and helped improve this article.

Contact the authors

Martin Hannigan
Boston, MA, USA
hannigan@gmail.com
+1 617 821 6079

Job Snijders
Amsterdam, NL
job@instituut.net
+31 6 54942365

Original post appeared on RIPE Labs.

Martin Hannigan is a builder of Internet networks and data centres.

Rate this article

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.