Seven years of running IPoE broadband in Nepal, and the outage that taught us more than any lab ever did.
The transition from IPv4 to IPv6 is usually presented as a protocol upgrade. In a lab, that’s roughly true. You enable the address family, you advertise a prefix, you ping something, you move on.
In a production broadband network, it is not a protocol upgrade. It is a subscriber-management project, a Customer Premises Equipment (CPE) interoperability project, a Layer 2 project, a logging project, and an observability project, and IPv6 happens to be the thing that exposes all five at once.
We started testing IPv6 at WorldLink Communications in May 2016. We went live in July 2019. Today the network carries more than a million Internet Protocol over Ethernet (IPoE) subscribers, 100% of them IPv6-enabled, with around 70% of our traffic riding IPv6.
This is what we learned in between. Some of it is architecture. A lot of it is scar tissue.
Why IPoE, and why dual-stack?
Traditional broadband ran on Point-to-Point Protocol over Ethernet (PPPoE). IPoE removes the PPP session: The CPE simply does DHCP, and the Broadband Network Gateway (BNG) identifies the subscriber from what the access network attaches to the request — Option 82 circuit-ID, remote-ID, VLAN, MAC, with RADIUS supplying policy on top.
That’s simpler for the customer and cheaper to scale. It also means something people underestimate:
> There is no PPP keepalive. The DHCP and DHCPv6 lease is the session.
Hold that thought. Half of this article follows from it.
As for dual-stack, we looked at translation and encapsulation, and chose dual-stack because it was the only option that didn’t put a translator in the path of every customer forever. IPv4 exhaustion was already forcing us toward Carrier-Grade Network Address Translation (CGNAT), and CGNAT is capex that grows with every subscriber you add — boxes, sessions, port blocks, and a legal obligation to log all of it.
Native IPv6 has none of that. Every packet we move onto IPv6 is a packet CGNAT never has to touch.
The important detail in Figure 1: IPv4 and IPv6 are two independent forwarding and service planes that happen to share one subscriber session. Treat them as one thing, and you will eventually debug them as one thing, which is where troubleshooting goes to die.
The addressing plan, and the one number I’d change
We received a /32 allocation from APNIC and structured it conventionally, as shown in Table 1.
| Purpose | Prefix | Why |
| Router Loopbacks | /128 | One address per node |
| P2P backbone links | /127 | No ND cache to exhaust |
| Server and CDN Segments | /64 | Keep SLAAC working |
| CPE WAN (IA_NA) | /128 out of a /64 | The address on the WAN Interface |
| CPE LAN (IA_PD) | /56 (**Adapted /64 per CPE) | The prefix delegated to the home |
The last row in Table 1 is the one I’d change.
A /64 prefix delegation gives the customer exactly one usable LAN. No guest network, no separate IoT segment, no VLANs behind the CPE, no room for anything they might want in five years. And a single /32 holds 16.7 million /56 assignments, so frugality here buys you nothing you will ever spend.
If I were planning this today, I’d delegate a /56, or a /60 as the absolute floor, and /48 for business customers. IPv6’s whole advantage is that you don’t have to be clever about address conservation. Using it like IPv4 wastes the only thing it gives you for free.
DHCPv6 is not DHCPv4 with longer addresses
This is where most dual-stack deployments start to hurt, so it’s worth being precise about the machinery.
Four objects you’ll be reading in packet captures for the rest of your career:
- DHCP Unique Identifier (DUID): Identifies the client. Not the MAC, not the IP.
- Identity Association for Non-temporary Addresses (IA_NA): The /128 on the CPE’s WAN interface.
- Identity Association for Prefix Delegation (IA_PD): The prefix delegated to the customer’s LAN.
- Option Request Option (ORO): Option 23 is the DNS server list, Option 24 is the domain search list.
The exchange is Solicit, Advertise, Request, Reply (SARR). Clients on User Datagram Protocol (UDP) 546, servers on 547, and all of it over multicast. RADIUS sits behind the BNG, handing down Framed-IPv6-Pool, the delegated pool name, and the policing profile.
And then there are T1 and T2. T1 is when the client renews with the server it knows. T2 is when it gives up on that server and rebinds to anyone listening. When the lease finally expires, the client throws everything away and starts again from SOLICIT.
That last sentence is the entire outage.
The renew–rebind cliff
Figure 4 shows a real sequence from our network. A fibre cut above the Optical Line Terminal (OLT) occurs, but the BNG never hears a thing, so as far as it’s concerned, the subscriber is still perfectly bound.
At T1 the CPE renews. Silence. At T2 it rebinds. Silence. At lease expiry it drops the address and the prefix and sends a fresh SOLICIT.
And this is where CPE vendors stop agreeing with each other.
Under RFC 3315, the client sends one SOLICIT with one transaction ID containing both IAs. The BNG sees one client renegotiating and recovers cleanly.
Under RFC 8415, a client is permitted to maintain a separate state machine per IA option, so it sends one SOLICIT with Transaction Identifier (XID) 1 for IA_NA, and another with XID 2 for IA_PD. Now the BNG sees two clients. Neither of them ever completes, because neither request is for a whole session.
Then the retransmit timers kick in, and nobody clears the old state.
We measured 116 SOLICITs in one minute from a single CPE. Project that across just 10% of an 80,000-subscriber BNG and you get roughly 15,467 SOLICITs per second. At that rate, your DHCPv6 control-plane violation counter is not going to clear — not today, not ever.
The DBOR effect
Before diving into the details, the failure sequence looked like this:
Delete-binding-on-renegotiation (DBOR) tells the BNG that a SOLICIT arriving while a client is already bound should tear down the existing binding and rebuild it. It exists for good reasons in DHCPv4. In DHCPv6, with a CPE that emits new SOLICIT XIDs whenever it feels like it, it does as shown in Figure 5.
The customer’s session gets destroyed and rebuilt, both IPv4 and IPv6, without the CPE ever being told. The CPE keeps using the address and prefix it already had. At the next renew the BNG answers with status code 13: NoAddrAvail (2) and NoPrefixAvail (6). The CPE tries again. Around it goes.
It only stops at a power cut or a fibre cut.
What that looked like from the operations floor:
- No IPv6 at all for every subscriber caught in it.
- CGNAT cards spiking, because all that traffic fell back onto IPv4 NAT.
- A call centre flooded with reports that Facebook and YouTube are slow (nobody ever calls to report a DHCPv6 problem).
The immediate workaround that actually worked — fitting an older router, because older firmware still speaks RFC 3315.
The vendor conversation went in circles for a while, and it’s worth describing its general shape. The CPE vendor was correct that SOLICIT-with-ORO-only is legitimate under RFC 8415, and that ORO-23 and ORO-24 are required for Broadband Forum (BBF) IPv6 compliance. The BNG vendor was correct that it was already returning both options in its REPLY. Both statements were true. The customer still had no IPv6.
The fix was never a knob. It was agreement. RFC 8415 section18.1, and RFC 7550 section 4 before it, both say a client should use a single transaction for all IA options on an interface, precisely so the client can keep one session and one state machine. Our recommendations ended up as:
On the CPE:
- Suppress ORO-only SOLICIT and REQUEST messages.
- Use one state machine for IA_NA, IA_PD, and ORO.
- Use a single exchange for multiple IA options in SOLICIT, RENEW, and REBIND.
- After a rebind failure, terminate older SOLICIT processes before starting new ones.
- Retransmit the same XID until SARR completes.
- Return NoBinding rather than NoAddrAvail / NoPrefixAvail when what you want is a client-side stack reset
On the BNG:
- Enable Option 24 ORO parameters if the platform supports them.
- Keep identical lease timers for IA_NA and IA_PD.
- Think very hard before enabling DBOR for DHCPv6 at all.
And the general lesson, which cost us more than any of the specifics, is that in dual-stack IPoE there is no keepalive. Persistent DHCP state for the full lease duration is the only thing holding the subscriber session together. Anything that deletes bindings out from under a client is playing with the session itself.
Most of our early failures were Layer 2
Before any of the above, we had a simpler problem: DHCPv6 runs on multicast, and our access network had been carefully tuned over years to suppress multicast.
Every one of those cost us real customer-visible time. The snooping one is the most seductive, because DHCP snooping genuinely helps you troubleshoot, until you’re running 8,000 subscribers on an OLT with a four-entry-per-CPE snoop table, mass power restoration churns it, and the OLT starts discarding the DHCPv6-REPLY or the DHCPv4-ACK on the way back down. Then you have idle CPEs and a beautiful, useless snooping database.
Leave the OLT doing pure forwarding and ranging. Snoop somewhere that can afford it.
The rest of the operational surface
Things that don’t get their own war story but will find you anyway:
Dual-stack shaping: One RADIUS bandwidth value becomes one dynamic policer and two filter sets — IPv4 in/out, IPv6 in/out. Get this wrong, and customers get double their plan, or half of it, depending on which way you got it wrong.
Security, because NAT was never a firewall: Bogon filtering on upstream routing policy. Control-plane filters that admit infrastructure prefixes only. Reverse Path Forwarding (RPF) on the gateway. Exploitable ports blocked. RA Guard and DHCPv6 Guard in the access layer. And the uncomfortable one: Giving a customer a globally routable prefix means their CPE firewall is now load-bearing, and CPE IPv6 firewalling was, and still is, in places, noticeably weaker than the IPv4 equivalent.
CPE diagnostic gaps: On IPv4 you can see which device got which lease. On IPv6, most consumer CPEs won’t tell you which end device holds which address, and many still ship without IPv6 ping or traceroute in the web UI. Your support team is debugging blind unless you plan for that.
MTU, end to end: ‘Google works but this one site doesn’t’ is a Maximum Transmission Unit (MTU) or Path MTU Discovery (PMTUD) problem far more often than it’s anything else. Validate the real path: Ethernet, VLANs, Multiprotocol Label Switching (MPLS), BNG, CGNAT, not the number on the CPE’s status page.
CGNAT logging and correlation: If 10.10.10.25:52341 became 203.x.x.x:49122 at 12:31:45, you need to be able to name the subscriber months later. That’s not a nice-to-have; it’s a legal requirement in most jurisdictions.
Design for the reconnection storm, not the average: The question is never ‘what happens when everything works?’ It’s ‘what happens when 100,000 subscribers reconnect in the same thirty seconds after a regional power cut?’ DHCP infrastructure needs horizontal scale, redundancy, lease-database resilience, and failure-domain isolation sized for that moment.
Build the data model, not the dashboard
At a million sessions, the question is never ‘which IP address is this?. It’s ‘which subscriber owned this address at this exact moment, and what else was true about them?’
We run the usual stack: LibreNMS, Grafana, InfluxDB, Telegraf, NetBox, ELK, and flow monitoring. None of that is the point. The point is correlation — one subscriber key that every other system can be joined on.
If answering ‘show me everything for this subscriber’ takes five systems and a spreadsheet, the network isn’t your problem. The data model is.
Two things matter more than the tooling choice:
1. Monitor IPv4 and IPv6 as separate KPIs — not the ‘Internet is up’. Track the DHCP success rate and DHCPv6 success rate separately. Measure prefix-delegation failures as their own metric. Monitor CGNAT session and port use on one side, and IPv6 prefix use and reachability on the other. Our DBOR incident was visible in the DHCPv6 violation counter long before it was visible in a customer complaint. We just weren’t looking at it as a first-class signal yet.
2. Size BNGs by the resource that runs out first. Not by the subscriber count on the datasheet. Track IPv4 and IPv6 sessions, DHCP and DHCPv6 transaction rates, Neighbor Discovery (ND) and Address Resolution Protocol (ARP) state, Forwarding Information Base (FIB) and Routing Information Base (RIB) entries, packets per second, and control-plane use. A BNG can sit at half its rated subscriber capacity and still fall over because neighbor-discovery state hit a wall.
The number that actually matters
When we went live in 2019, we had about 100,000 IPv6-enabled customers out of 290,000, and IPv6 had just crossed half our traffic. Today it’s 100% enabled and roughly 70% of traffic on a network more than three times the size.
Those two percentages are not the same kind of number, and the gap between them is the whole story.
A subscriber with an IPv6 address and a broken prefix delegation counts as enabled. So does one behind a CPE that never successfully renews. Enablement is what your provisioning system believes. Traffic is what actually happened. Bytes can’t be faked, and every point you move from IPv4 to IPv6 is CGNAT capacity you never have to buy.
So don’t report IPv6 adoption as the percentage of subscribers holding an address. Report the percentage of traffic successfully using it. The first number makes for a nicer slide. The second one tells you whether it works.
What I’d do differently
Observability and automation from day one, not as a phase two that never quite arrives. Everything in section 8 was retrofitted, and retrofitting correlation across a live million-subscriber network is considerably harder than designing for it.
A CPE interoperability lab before mass deployment. Not a reference CPE, but the actual devices your customers actually have, at the firmware versions they actually run. Every failure in sections 4 and 5 was a CPE-behaviour failure that a lab with the right two routers would have caught in an afternoon.
Compliance agreed in writing between CPE and BNG vendors, naming the RFC, before either box is deployed at scale. ‘Both ends are compliant’ is not the same as ‘both ends are compatible’, and you find out which one you have at 8pm on a Friday.
And a /56. We chose /64 due to CPE Limitation.
The takeaway
After seven years of this, the thing I’d most want to hand to someone starting out is this:
- IPv6 deployment is not primarily an IPv6 project. The protocol is well understood and thoroughly documented. The hard part is everything it touches: Subscriber management, DHCP, DHCPv6, RADIUS, the BNG, CGNAT, CPE firmware, Layer 2 access, security, logging, monitoring, and the support process that has to explain all of it to a customer who just wants YouTube to load.
- IPv6 works at massive broadband scale. We have a million subscribers as proof. But it works because of the system built around it, not because somebody enabled an address family.
- Don’t measure it by how many subscribers have an IPv6 address. Measure it by how much real traffic is successfully using one.
That’s when IPv6 stops being a project and becomes part of the network.
Kijush Maharjan is the Principal Network Engineer at WorldLink Communications and the founder and Secretary of npNOG. He is recognized as an IPv6 pioneer in Nepal.
This article draws on operational experience at WorldLink Communications, Nepal. Specific implementation details have been generalized for confidentiality and security reasons.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.