Telekom Malaysia (TM) got its first IPv6 subnet allocation back in 2004. At that time, I believe we were still running the “tuut tuut beep beep …” and five minutes later the dial-up modem would connect. Now we are running two different Autonomous System Numbers (ASNs) to carry a dedicated Internet service and VPN.
To quote what Dr Phillip Smith said in an IPv6 deployment training session, “You can’t just wake up one morning, grab a cup of coffee and decided to enable IPv6 for the whole infrastructure. It doesn’t work like that. You need detailed planning and validation first.”
He was right. For TM, IPv6 is a ‘journey’. This post will explain how TM delivered IPv6 inside the network, a process that took substantial effort over several years.
Infrastructure node setup
In 2004, TM was running OSPFv2 for the Interior Gateway Protocol (IGP) to carry all the infrastructure routes. With all the heavy lifting required to either enable OSPFv3 or migrate to another IGP, TM decided to take this opportunity to fully modernize the network. Build planning and preparations for the new infrastructure began and one of the top requirements was IPv6 support across the network. By 2008, extensive testing to qualify the new infrastructure had been completed, and from day one, it would be built to run the IS-IS routing protocol.
Together with the MPLS network, which combined both the legacy Internet and VPN network, TM also enabled IPv6 Provider Edge (6PE), which allowed TM to carry the IPv6 route via the Link State Packet (LSP).
While it’s not the first choice nowadays, we opted for 6PE because the changes were manageable with available resources and suited TM’s requirements at the time. There were also some perceived challenges to going fully native.
Firstly, it was cost. Not all TM equipment was able to support native IPv6 transport so there would have been an additional cost to procure equipment. Another constraint was the content itself. Because most content was IPv4, we needed to invest in NAT64 and DNS64. TM’s main service is as an Internet Service Provider, therefore, Customer Premises Equipment (CPE) can remain unchanged for a longer period compared to mobile providers. However, the cost to upgrade both networks and CPE would have to have been carried by the user. This was unacceptable.
Secondly, tunnelling wasn’t being considered due to the large amount of traffic that needed to be managed. TM preferred to have complete control with minimal effort in managing the traffic for future growth.
Once we had that covered, we would need to bring in the IPv6 content.
To achieve this, by early 2011, we were happy to organize a dual-stack connection for every upstream and peer with which we interconnect — as long as they support IPv6. We didn’t stop there. Getting the content is a start but we wanted to deliver faster content, so TM approached the ‘giant’ content providers and invited them to position their servers already scattered within Malaysia inside TM’s own Server Farm. With all the planned tasks now completed it was time to bring in the customers.
Access node setup
Access is a critical point in delivering IPv6 reachability to the customer. TM had a mix of CPE being deployed within Malaysia, even with multiple revisions of the same model. Identifying which deployment method to pursue was a huge task in terms of cost and deployment strategy. After much investigation, we chose the path of least resistance — to begin with, what we had in front of us at the time.
Together with validating the infrastructure, another dedicated team was focused on testing the new dual-stack CPE in 2010. All the new CPE being acquired had to be able to support IPv6. Once we had that covered, all of the CPE already deployed and in production had to be sampled and brought back into the lab for detailed validation. The technical specification wasn’t enough to isolate which CPE could be firmware upgraded to support IPv6; it needed to be tested and equipment that wasn’t compatible needed to be physically replaced. It was a long process, but by the end of 2012, equipment testing and infrastructure planning were completed.
By then, we had decided to commercially roll out all CPE that had passed validation testing to new customers. It was delivered staggered, with CPE that required a firmware upgrade to existing customers. The firmware was upgraded remotely via TR-069. For customers who required new CPE, TM sent a technician to the customer’s home/premises for batch replacement.
The next couple of years were dedicated to rollout and expansion. During this period, plans were executed to update CPE firmware. We covered regions and targeted pools (based on subscribed user bandwidth) by phase. We rolled out in phases to ensure that any rising issue could be isolated, and the team could roll back if necessary.
Some good news arrived in 2015 — the Malaysian Communications and Multimedia Commission (MCMC) issued a Direction (PDF) that drove all Malaysian ISPs to provide dual-stack IP addresses to the end user (Figure 2). This inspired a steady increase in Malaysian IPv6 growth (Figure 3) and TM had already made these changes.
Onboard with other leading telcos
After many years of hard work and dedication by all team members, TM is proud to be among the leading ISPs for the global IPv6 deployment effort, as measured by the Internet Society (Figure 4).
IPv6 deployment advice
Based on TM’s journey deploying IPv6 in our network, there are four key attributes we can share that have proved successful.
Awareness: This is arguably the most important first step to successful IPv6 deployment. It’s not just the network engineers who need to support the initiative to deploy IPv6. Personnel at all levels must play an important role to ensure deployment will be successful. We invested in training from the bottom up on IPv6 awareness and found it to be an absolute requirement.
Leadership support: With awareness throughout the organization, we were able to achieve the support of leadership, which guaranteed end-to-end deliverables were made the priority. Top leadership encouraged inter-division support on the importance of IPv6 and enabled us to future-proof our infrastructure.
New growth and expansion: When TM began the journey to enable IPv6 inside our network, we made IPv6 support compulsory for any new node deployment. To validate those new devices, TM tested all equipment in our lab to ensure what’s written in the specification is accurate and able to operate with multiple vendor nodes.
Security considerations: Introducing new capabilities inside the network comes with pros and cons. On day one, our security protection for IPv6 was not as strong as it is right now. We continue to learn and improve our control plane regularly. We learn from mistakes, continued education in best practices, and gain experience from attacks. We also look outward and learn from issues with other ISPs. We strive to ensure no security blind spots in our deployment of IPv6 services.
Defeating challenges to IPv6 deployment in TM
Multi-vendor equipment
To avoid dependency on a single vendor and major impact/downtime due to software bugs, TM is built upon and continues to grow in a multi-vendor environment. Yes, we manage to avoid most issues but there are challenges in supporting new technologies where each vendor can have a different way of doing things. To mitigate this, we stick to the common guidelines — follow the RFC.
Legacy and new infrastructure
ISPs cannot be rebuilt from the bottom up with every tech refresh. Although TM deployed a new high-end node, we still had to maintain the legacy network because deploying and migrating at the same time would require massive manpower and budget. TM studies whether the legacy node can support IPv6 with just an OS upgrade or if it requires new hardware. We prioritize nodes that can upgrade. For nodes that require hardware upgrades, we migrate the services into a new node.
A massive number of devices
TM is one of the biggest ISPs in Malaysia with millions of users. Managing this amount manually would stretch our resources so we opt for automation where possible. In this case, we pushed new firmware into the CPE and upgrade remotely via TR-069. This allows mass automation, reducing the time and cost to deliver the latest features to our users.
Regulatory bodies
When TM started its IPv6 journey, buying new CPE that supported both IPv6 and IPv4 was much more expensive than native IPv4-only, and the cost to buy equipment must be added to deployment expenses. Regulatory bodies that mandate IPv6 deployment can help keep equipment costs low.
IPv6 content
When you first start offering IPv6 service delivery, much of the content is still IPv4-only. This means that users don’t see any immediate advantage — they either delay fetching it via IPv4 over a fallback — because the Happy Eyeballs (RFC 6555) algorithm preferences an attempt over IPv6 — or they get it via IPv4 and don’t notice any improvement. With 40% of users now accessing Google via IPv6, TM found that by preparing its infrastructure path to peers and improving IPv6 connectivity to the content, more dual-stack accessible content becomes visible. Because this reduces pressure on the IPv4 CGN, it can improve overall service, albeit at a slightly increased connection cost.
Human nature
This one is tricky. We saw our users intentionally disable IPv6 and we assume it’s because they read somewhere that with single stack IPv4, the connection will be much faster. This behaviour blocks CPE support for IPv6. To overcome this we modified the user-configurable landing page to only show the required information and disabled the rest. It was a simple solution that solved the issue TM faced at the time.
Security measures
Security is an ongoing effort. We must take all security measures available to us to protect the control plane via IPv6 and the forwarding plane. While similar to IPv4, it’s not exactly the same, for example, IPv6 has more dependency on ICMPv6 compared to ICMP in IPv4. We learned how others are doing it from the APNIC Blog and APNIC Academy training and use this knowledge to implement best practices in our network. We don’t necessarily have to have the same experience to learn from it.
Watch Muzamer’s APNIC 54 presentation on Telekom Malaysia’s IPv6 deployment experience:
Muzamer Mohd Azalan is a network engineer with eight years experience managing the core network for one of the biggest ISPs in Malaysia. He is currently the Manager of Network Service IP Operations for Telekom Malaysia. He also shares his knowledge and experience as a volunteer community trainer for APNIC.
Begin, or continue, your IPv6 journey here:
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.