The ISP community in New Zealand is a bit different to other economies.
Most of the last-mile fibre and copper access network is provided by wholesale-only operators who are subject to various forms of regulation affecting price, quality and other conditions. Retail Service Providers (RSPs) typically deliver Internet services directly to end users by leasing an access service from one of these wholesale operators.
Chorus is the largest wholesale operator with a national network (see Figure 1) that provides about 70% of the fibre access network and 100% of the copper access network (an artefact of its incumbent past).
We have 1.2M broadband connections with most of those (681k) fibre Gigabyte Passive Optical Network (GPON) and the rest copper VDSL or ADSL; about 1,000 GPON Optical Line Terminals (OLTs) in 410 sites and 7,500 DSLAMs in 6,300 sites are backhauled with 210 microwave radio nodes and 520 Dense Wavelength Division Multiplexing (DWDM) nodes. The RSPs’ Broadband Network Gateways (BNGs) are interconnected with the OLTs and Digital Subscriber Line Access Multiplexer (DSLAMs) using a Core Ethernet Network (see Figure 2) of 350 nodes.
The connection between an end user’s premises and their chosen RSP is an Ethernet VLAN (see Figure 2). In Metro Ethernet Forum (MEF) terms this is an Access EVPL service (MEF 33 section 6.2).
To ensure there is always enough capacity, we predict and monitor aggregate traffic demand and ensure the build program stays ahead of the demand.
Chorus’ commitment to New Zealand
Chorus was the first New Zealand organization to formally commit to ensuring the capacity of our network exceeds demand, now, and well into the future.
“We agreed with the Commerce Commission and Crown Infrastructure Partners that data traffic through our core network supporting copper and fibre broadband at any one time must not exceed more than 95% of network capacity.”
In practice, congestion free means no link will reach 95% utilization during any five-minute interval. It’s a simple way to show we’re not dropping any traffic. A summary view is shown in Figure 3.
Is 95% link utilization really congestion free? Well… no, but it’s close enough. Figure 4 shows some observations from two 10G links in our network during a fault back in 2016 (the last time we had a fault long enough to get a useful range of data). Packet loss did not exceed 0.2% at 95% utilization which, while not zero, is still really good.
See the technical white paper for more detail about congestion free (note: since the paper was written we have moved from 15-minute to 5-minute intervals).
Aggregate traffic demand and Rugby World Cup 2019
The ISP community in New Zealand faced one of its biggest tests in October 2019 with the Rugby World Cup (RWC). All games were streamed live with some of the key games also broadcast free to air. While aggregate traffic through the Chorus network has been increasing with a gentle exponential trend, the RWC had a substantial impact, which you can see in Figure 5.
You can also see that ‘peak Copper’ occurred about June 2018 and is now in steady decline.
Figure 6 shows a close up look at the RWC period, which successfully remained congestion free. The top three peaks were the quarter finals (they were not free to air) and the unrelated Fortnite update. Epic Games releases updates to their Fortnite game at the same time across the globe and that just happens to be peak evening viewing time in New Zealand.
TCP is greedy so is congestion free even possible?
Well, our observations show yes it is, but why?
There are probably two things at play here:
- The peak demand by far is in the evening (see Figure 6) when people are watching video. Video is rate controlled by the apps and content being streamed (for example, a YouTube or Netflix client and the coding of the content) rather than TCP as such.
- TCP adapts to the bottleneck and we are ensuring that our network is not part of the bottleneck, that is, it’s somewhere else such as the content cache or IX.
Chorus has regulatory and contractual commitments to the government that include service performance as well as price. Specifically for GPON fibre connections, there are packet level performance expectations for the two traffic classes: High Priority (HP) and Low Priority (LP) as marked with the 802.1p PCP field (Table 1).
| Class | Loss | Round trip | |
| Delay | Delay variation | ||
| HP | ≤ 0.1% | ≤ 5ms | ≤ 1ms |
| LP | ≤ 2% | none | none |
Table 1 — UFB1 contract commitments with Crown Infrastructure Partners.
These parameters apply for each five-minute period, and any period that does not meet the criteria is considered ‘down time’ with financial penalties. Chorus operates a set of reference probes, one per OLT, that perform 10 tests per second to verify compliance with the performance targets.
Even though it’s regulated as a near-monopoly, Chorus faces competition from a DOCSIS cable network and ‘fixed wireless’ from three cellular network operators and several smaller local wireless access providers. By being congestion free, we avoid penalties, build trust and become the preferred network for end-users.
Adapted from a talk given at NZNOG 2020.
Richard Read is a Principal Solution Architect with Chorus in New Zealand
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.