Expired certificates halve IPv6 capability in Bhutan, Sri Lanka

Last year when I was in Bhutan for btNOG 5, Bhutan Telecom (BT) spoke about IPv6 deployment on their mobile network. As the incumbent operator, it significantly increased the economy’s IPv6 capabilities to around 12%.

It was a good news story, and one that I personally was proud to point out to operators in more developed economies given I was a part of the team who deployed IPv6 in BT’s IP backbone.

Then in January this year, I was asked by the SANOG 33 Program Committee if I would submit a presentation on IPv6 in the region and highlight the host economy’s recent efforts. Of course I accepted, but upon returning to the stats I found something had gone totally awry in the past two months, which as we would find, was not contained to Bhutan.

Figure 1 — Bhutan IPv6 capability, June 2019. You can see the dip at the start of 2019.

Diagnosing the problem

Before working at APNIC, I was a network engineer at BT where we would continually be troubleshooting issues like these. So, I contacted my old colleagues at BT and asked if I could come a few days before the SANOG conference to work with them to understand what had caused this dip in capability.

Our first clue became apparent before I even arrived at their offices. Upon landing I got a local SIM card (4G enabled), checked my prefix — IPv6, great! — and headed to my parents’ place for the evening. The next day, before meeting with my BT colleagues, I checked my phone again and found I no longer had an IPv6 prefix. Hmm.

I toggled the airplane mode, essentially resetting the connection, and voila, I got my v6 prefix back! But, I quickly lost it upon receiving a text message from one of my colleagues. Again, I toggled, and I got my IPv6 prefix back. The same thing happened when I received a phone call.

After discussing the issue with the mobile team at BT and realizing it was not a handset problem, we figured out that because BT did not implement Voice over LTE (VoLTE) — given Bhutan is a small economy with not many Internet users, voice and SMS over LTE was too expensive — SMS and voice calls were falling back to 3G, which was missing an IPv6 profile.

After an hour of configuration and updates by the B-Mobile team, BT and Bhutan’s IPv6 capabilities quickly increased.

Problem solved…or so it seemed

A few days later, during the IPv6 deployment workshop that I was facilitating with Philip Smith at SANOG, I shared the case study with the class, noting the importance of needing to continually monitor your network for such issues.

It piqued the interest of the participants who started looking at their IPv6 capabilities, one of whom recognized a similar pattern in his economy, Sri Lanka. The likeness of the dip (15% down to 7%) was uncanny.

Figure 2 — Sri Lanka IPv6 capability, June 2019.

I asked him what changes they had made to the network in December, to which he said they had deployed VoLTE sometime in December 2018, after which their capability was back above 15%.

Root cause?

Upon digging a little deeper into the issue, it turns out it may have been due to an issue with an expired Ericsson certificate — an issue that also caused extensive data outages. Most operators who were running Ericsson’s mobile core had to roll back to an older version in December 2018 upon which the IPv6 profile on 3G was removed.

Again, it’s instances like these that I use to remind engineers of the importance of continually monitoring and measuring your network so you can recognize if unforeseen issues or recent changes to the network have had any adverse effects. Not only does this save you time troubleshooting such problems but it means your network is performing the way it should be.

Rate this article

---

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.