[Podcast] Taking IPv6 by the horns

With more than 420M mobile users and 260M cable users, Reliance Jio has grown to become one of the world’s largest telcos in less than a decade.

Mr Nagaraj, Senior Executive Vice President at Reliance Industries, has overseen this growth from the network and operational side. He credits this unheard-of feat to Reliance Jio’s bold strategy to target and dominate the mass market by offering a cheaper and more modern service than their rivals, and rolling out IPv6 across its network. The latter achievement has catapulted both India’s and the world’s IPv6 capabilities in the last five years.

The following is a transcript from a PING! podcast featuring Mr Nagaraj and APNIC’s Chief Scientist, Geoff Huston, who has been charting this growth on APNIC Labs and was keen to understand how Mr Nagaraj was able to convince Reliance Jio to adopt IPv6 at a time when it was not fully supported.

Note: This transcript has been edited for clarity and brevity. Please refer to the podcast for the full discussion.

Geoff Huston

So, this is an amazing achievement and certainly India today has the largest national pool of IPv6 users on the planet. And Reliance Jio was there from the start. So, let’s talk about that start.

It’s 2014, and you’re getting your plans ready because from my data what I see is some experiments in 2014, initial deployment in 2015, and then you really tuned it on in early 2016.

Now that’s some time ago. And at the time there wasn’t much IPv6 about. Most of the large corporate services, the Googles and Microsofts and so on of this world, were IPv4-only and we’re just toying around with IPv6. The handset manufacturers, be it Apple or Android systems, were basically concentrating on the IPv4 market because that’s where the money was. So, there wasn’t a lot of attention on IPv6.

So, this decision by Reliance Jio seems very brave. How did you sell it to your management given that most of these companies are very conservative? When I worked for one, I was completely unsuccessful in getting them to get rid of their Synchronous Digital Hierarchy (SDH) and go to an all-switch network. How did you succeed in convincing them that the time was right to make its core IPv6 and treat IPv4 as the overlay back then?

Mr Nagaraj

One of the good things that happened to me was management knew [and trusted] my capabilities.

In 2000, when I had just joined Reliance Industries Limited, my challenges were to [show management] how having an IP network was important, how IPv6 is important, and how IP MPLS is important. It took me two to three years to convince them that this was the required approach. During this time, I also helped successfully launch Code-Division Multiple Access (CDMA) data connectivity. At the time, it was quite a minuscule [project] but it created certain hype in the industry. Since the confidence was there, I got quite good support from my management. That’s the first thing.

The second thing is we took stock of all our devices — from the SIM manufacturers, from the equipment manufacturers, as well as the application people — and made IPv6 support as the fundamental purchase criteria. For vendors that did not have this, we gave them a timeline to move to support dual-stack. And we supported them for the transition period of six months to one year.

When we started rolling over the network completely to IPv6, we first transitioned the management plane to IPv6. Then we transitioned the control plane, wherever it was possible, to IPv6. The user plane, from day one, was on IPv6. Since the user plane was completely on IPv6, we achieved almost 80% of our IPv6 deployment on day one.

Geoff

So, you didn’t go dual-stack in your radio network from the start? You went IPv6-only?

Nagaraj

Yes, completely on our radio network and entire IP ecosystem interconnections, they happen on IPv6. For certain applications that could not support IPv6 — for example, Policy and Charging Rules Function (PCRF) and Diameter Routing Agent (DRA) — we hid them in the core on IPv4. But for the user plane, they’ve always been talking via IPv6.

Geoff

So again, I’ll go back to that time: There weren’t a lot of big rollouts that had IPv6. And that meant that the vendors weren’t that interested in fixing bugs in their code. It was just kind of ‘no one’s complaining, why should we fix this problem?’

What was your experience with your vendors who were supplying IPv6 equipment at that time? Were they up to scratch? Or did they need a certain amount of encouragement to improve their code?

Nagaraj

You are very right. Vendors were not ready to do it. But when we had made it a purchase criterion, and continuously worked with them, we started to see change.

New technologies like cloud would still like to do it only on IPv4. And many of the vendors at the moment — which we are working with on our 5G rollout — all the core elements are only on IPv4. Because, when they try to roll out a new technology they tend to develop them to work with proven technologies. With IPv6, they see challenges. So, they would like to attack new technologies with their much-proven IPv4. And only when those things are working, then they will go for IPv6.

If there is a real requirement of the network business to support IPv4 then we will support it, but we give them a timeframe to support dual-stack, not IPv4-only. When we initially had these conversations it was difficult. But then when we proved that Reliance Jio can reach a mass market, the vendors realized that if this was to be a game-changing moment, they would like to participate. For example, Samsung did our entire radio and core network on IPv6. Then we got support for our IP Multimedia Subsystem (IMS) stack. Then for video.

DNS was another interesting example. When users make a query, they make a DNS resolution first. I will send only an AAAA response first so that the user will get IPv6 as a first response. If that website is not supporting IPv6 then they’ll use IPv4. So, we made the network in such a way that the people will be forced to use IPv6 as the first choice.

To be frank, it’s not been an easy journey. The journey has not ended, but I think we’ve made waves and shown to the world that this is possible.

Geoff

You talked about IPv6 in the DNS. And in late 2018 – early 2019, Reliance Jio turned on DNSSEC validation in its DNS resolvers. Again, this is unusual in today’s production networks. DNSSEC is not widely implemented outside of Google’s own service. And it’s quite unusual for an ISP-based resolver to turn on DNSSEC. I think Reliance Jio and Comcast in the US and very few others have done this. Personally, I think it’s a great thing, understanding truth in the DNS is important. But what was behind your motivation to turn this on? Why did you do this?

Nagaraj

I feel that we should protect our DNS as much as possible. I can confidently say that we have made our DNS as robust, as secure, as capable as it can be. And we had to bifurcate this particular thing also for a separate DNS system for the Internet and a separate DNS system for our network elements, for our FTP. So if anyone causes disturbance they won’t cause a disturbance in other parts of the network.

Robbie Mitchell

I’d like to return to your IPv6 efforts, Mr Nagaraj. The sheer size of Reliance Jio’s user base has helped it to really push the need to support IPv6 among your vendors, but has it had any effect on content providers and more established enterprises who feel the need to still hold on to IPv4?

Nagaraj

Today we have a customer base of nearly 500M, which may grow to nearly 600M in the next two to three years. And we have 20M IPv4 addresses. 20M for all enterprises, all fibre-to-the-home (FTTH), and all mobile customers. We have only been able to manage that because we are using IPv6.

For content providers, we consulted the top 20% of the websites across the different industries and started working with them to make them at least dual-stack. The top 20% of services account for about 80 – 85% of our Internet traffic. And that 20% we changed. Of the 80% that remains on IPv4, let them remain because they are only responsible for about only 15 – 20% of our Internet traffic.

We also worked with the banking industry, which is among the top industries within the economy. When I told one very prominent bank in India that we worked with to make them dual-stack that an IPv6 website will not have all the facilities of an IPv4 website, they said ‘that’s a problem, I’ll get customer complaints’. The other problem was when I said that an IPv6 website is susceptible to crash because it will be dealing with more hits. In these instances, you have to work with people and try to not make them scared.

It has not been an easy journey but we are trying to make those sites — as much as possible — also convert to IPv6. It’s a long journey and it continues. I don’t know whether we’ll get 100% there but we are seeing up to 90% of our traffic transit via IPv6.

Geoff

There are some interesting issues around that and one of them is the outage of Slack late last year. Ultimately, the reason why Slack was offline for 24 hours was a complex DNS issue relating to DNSSEC and Route 53. But the basic cause — they were IPv4 only. And because users who are dual-stacked prefer to ask in IPv6 first, they were getting the wrong answer in IPv6 that was telling them ‘there was no Slack, nothing’. And so, no one could reach Slack because they were IPv4-only. And while the original requests were for AAAA records — the IPv6 records — Slack just died for everybody.

So, it’s important to convince these folk that IPv4-only is sometimes a liability if you haven’t got everything in place working perfectly. These days, no one does the job themselves, or by themselves. We all rely on the Route 53s of this world, the Akamais, the Cloudflares, and so on. This means not only do you have to get your own job in order but everyone else has to as well.

And as you were saying with banks, I noticed in Australia, there is not one bank that operates its own infrastructure anymore for its retail banking. Every one of the big banks is now on a cloud. And trying to persuade the cloud folk that dual-stack is important, as you’re saying, is very difficult.

Nagaraj

Most of the popular cloud services are still on IPv4. All the cloud applications are still on IPv4.

Geoff

This is such an issue.

I’m looking around at the region and India is this island of concentrated IPv6. Bangladesh? Not really. Nepal? No. Even Bhutan? Not really. But it’s worked for India and yet one would have thought the technological landscape should be no different. Why do you think your neighbouring economies such as those three haven’t done the jump themselves?

Nagaraj

The thing that changed the situation in India is that Reliance Jio entered the market. And Reliance Jio took a strategy to target the mass market.

Because we entered as a greenfield, we entered completely as an IP network. We did not allow any other legacy technology. And so we embraced and boldly pushed IPv6 and made it successful. Now I see other operators in our society trying to follow in our footsteps. Originally 2G and 3G remained for a long time. 4G was adopted only when we made it our choice.

In 2010, there were 45 to 50 deployments of WiMax happening all over the world, but only for broadband Internet access. We had a choice. The WiMax chairman came and met us to discuss adopting WiMax. We said no, we should take a longer strategy and that’s why we chose lP. We adopted that technology, made it successful, and now today you can see there’s no 2G or 3G in India. All other operators have been changing the spectrum also to 4G. That means there has to be one leader who is required to take the bull by the horns and then push the technology down their throats to make it successful.

Today, other than some in Myanmar, there is no WiMax story anywhere in the world. Everybody changed to LTE. That is because it became a trendsetter in that particular thing not only in the technology selection but also on IPv6. And that is why India is leading the way with IPv6.

Geoff

There’s one other thing that Reliance Jio did that I’m curious about. Every economy auctioned the radio spectrum and, by and large, in every economy the price of the spectrum was much the same — it’s the same industry all over the world, they all bid at much the same rates. Yet when I look at the price of mobile data, India is cheaper than anyone else, and it’s fast. You have a product that is outstanding for the customer, it’s cheap and it’s fast. In America, I’d pay 10 times as much per gigabyte as I would pay if I was a customer in India. 10 times!

So here you are with the same kind of technology, blindingly fast service, but it’s so cheap. I’m interested in how this is achieved because when you ask your counterparts in other economies they say ‘Well we bid so much for the spectrum, there are a lot of costs involved, we can’t make it any cheaper.’ Yet you’re saying, by your very actions, it can be cheaper. What are the elements that give you a product at this phenomenally low-cost point? How do you survive?

Nagaraj

Again, I’d like to thank the management for making a bold decision because they had a vision and their vision was not to target a small segment of the market but go after the masses. When you do a mass deployment strategy you can keep the prices low and recover the losses through the sheer number of users.

Also, Reliance Jio is different compared to many other companies that want returns immediately — If I invest one rupee today, I want two rupees tomorrow, otherwise, I don’t want to invest. Our organization is not like that. Instead, we invest more like you would in public infrastructures, such as roads. You will not get a return from this on day one, it takes time. In this respect, we actually got support from the government.

What we have achieved in five years, there is probably no history of this happening before or whether it can be duplicated again. These are all factors as to why our neighbours have not achieved what we have.

Geoff

Well, it’s so evident that they’ve not done the price and they’ve not done the technology the same way. Absolutely not.

Nagaraj

Hats off to my management for that. That management vision.

Robbie

Starting from scratch though, do you feel you had an advantage over all incumbents given that, as you said, you didn’t have any legacy technologies to deal with?

Nagaraj

In the sense that as a greenfield we knew we had a great advantage. Of course. As I said, certain bold decisions were required. Certain managemental support was required. We knew that this was going to succeed.

And when we have network problems, we don’t treat them as temporary system errors, we work to fix the issues permanently. Then we move forward, and we take on more and more new challenges.

Change is something that is inherently there. So, accept the change and accept the challenge. My management always puts stiffer and stiffer targets. If you want to achieve 100% you must have 300% as the target. And even if you achieve 150% you are still ahead of the 100% target.

Geoff

Legacy is a big issue. In the old 3G networks, the vendors used to charge by the protocol, by the minute. So, if you ran a dual-stack network on 3G you paid the vendor of the radio equipment twice as much as if you ran one protocol. That’s why everyone ran IPv4.

When they started to deploy 4G they didn’t deploy it to throw out 3G, they did it as a rolling sort of upgrade in various parts of the network. And to support 3G they had to keep IPv4 running internally as the only protocol and then everyone was faced with running IPv6 as an overlay and that’s why you saw all of this mucking around with encapsulation technologies and so on, which tried to sort of push the two together in very uncomfortable ways.

From my understanding, this was more similar to the Verizon story: You entered the market late; you have no 3G legacy to accommodate; no mono-protocol problem; and you can think about how you wanted to do it, not how does history force me to do it. This is part of Reliance Jio’s story that I’m hearing. And maybe sometimes it’s worth it to come into the market with a bold vision late knowing what you want to do, rather than come in early and then try and figure out the problems on the fly.

Nagaraj

I agree with you.

Geoff

Let’s talk a little bit about this growth in volume because this is the bit that interests me about your next engineering challenge. Obviously, you’re deploying 5G. And with the amount of content constantly rising, what we’re all finding is we’re hitting some real issues about getting that much traffic from the data centres where the Content Data Networks (CDNs) have their points of presence across to the end customer. And your internal engineering is now all about doing that last-mile access flow.

Now, it seems to me that the next conversation is about getting the CDNs to host their content inside your network, not just in major population centres. And to do so in such a way that they’re close to the points where there are customers to feed it so that the distance content travels, even within your network, is minimized. How’s that conversation going? Are they enthusiastic or reticent about working inside your network with their content?

Nagaraj

When I started my journey in Reliance Infocomm in 2000, 95% of the data was going outside the economy. So, 95% was going outside the economy, and only 5% remained in the economy.

Today, we have about 93% of the content staying in the economy. My international lines team consists of only seven people, which we are trying to reduce to five. How have we achieved this?

Today, we have 70 data anchoring point locations for our mobile network. And we have approximately 800 points of presence for our Layer 1. Whenever one of these points sees 200GB of traffic, we’ll install content caches like Google and Facebook in that location. When it reaches around 500GB, we’ll install caches like Akamai and Amazon. And wherever it is 1TB we’ll bring other peering locations, other caching locations. So we keep it dynamic and change it based upon the network and customer growth and customer requirement.

Geoff

So 95% of your, if you will, ‘sold traffic’ is domestic. It came from somewhere within India and is going to a customer in India.

What’s your objective for in-network over the coming years? In other words, ‘I didn’t even pull it from a peering point, didn’t pull it from anyone else, I pulled it from caches inside my network and delivered it to my customer.’ What’s your target for that?

Nagaraj

That’s an interesting possibility. By 2025, I expect our 5G FTTH environment will have 500,000 points of presence which will be completely dictated by real network requirements not just because of the locations we cover. There will, of course, be strategic locations outside of cities but within metro areas, we’ll go by the exact traffic and use those 200GB, 500GB, 1TB or greater thresholds.

You know when we started the Internet, people said ‘One location is sufficient. You put your PC in one location of the economy and others can connect via radio.’ Soon one location became two, two became five, five became 17, 17 became 43, 43 became 70. When the traffic starts increasing, you have to start putting locations closer to the edges.

Geoff

So if I was a big streaming provider, naming no names, and I was looking at the market of your customers then it seems to make sense to me to get very fast service without stressing the network I’d want to have my content cached in 800 locations. Eight hundred.

Nagaraj

It is definitely possible.

Geoff

Petabyte disc systems are neither big nor expensive. So will you help me build that? What’s the relationship to expand those points of presence to such a level that they’re part of your network as distinct from just me having lots of CDN points of presence? That this relationship is then different because where these caches and where these front ends go is part of the network’s engineering, not part of the CDN in isolation. Is that the next few years of your life?

Nagaraj

You know one of the success stories of Reliance Jio is not just managing communication wires. In 2010, when we entered the market, there was a stage where content providers dictated the market and service providers were just managing dumb pipes. But today, you see that the service providers have come to stay. Why? Because we work in an ecosystem. Now we are partners of Google, partners of Microsoft. We are a part of CDNs’ strategies. We also have our own CDNs, which we’re expanding.

Having said that, from the outset you must start seeing the value. What is the value in the Internet market? One is content, the other is eyeballs. So we have the eyeballs. The content providers don’t have the eyeballs. So who is the bigger of the two? Who leads the bigger pipe of the two? Ultimately, we’ll see the survival of both which the market will likely dictate.

Geoff

Oh, we’re all winning. I’ll take it then that I’ll get some help with my streaming network from Reliance Jio in terms of that placement, which seems to be a very productive way of doing things. So yes, you are looking at a very busy few years coming up to do this, but it should be a lot of fun.

Nagaraj

I told you that every day, everything is a challenge. You must stay ahead of the curve, think ahead, so you don’t get consumed by this progress. We were greenfield about five years ago but now we’re no longer greenfield. We are as brownfield as anybody else. You must keep ahead of the curve, otherwise, you will not survive.

Geoff

I think you’re in for a very exciting trip and I envy you.

Nagaraj

I agree with you.

Robbie

I have one last question. What was the biggest challenge you’ve had thus far with your IPv6 strategy that other providers can learn from?

Nagaraj

The biggest challenge I’ve had in this entire experience was private IPv4 space. We have been able to dictate to our vendors the need to support IPv6 but if a customer comes to us we have to accept them and ensure that they can connect their mobile device whether it supports IPv6 or not.

We first deployed a dual-stack to help with this. We worked with the four major SIM vendors to support IPv6 and have worked to convert our customers’ older SIMs. And we worked with mobile operators to support XLAT. Because we have about 20M IPv4 private address space and we have to support about 500M customers, how do we do it with a dual-stack mechanism? IPv6 is how.

In six months we’ll completely be using XLAT. Private IPv4 spaces will be confined only to customers’ mobile handsets. XLAT will convert that to IPv6 so in our network, only IPv6 will flow.

We were the only ones who faced this particular problem in the entire world because nobody had our customer base on an IP network. Even though we wanted to do IPv6, we had to accommodate for IPv4 until we had a strategy, in XLAT, to transition completely to IPv6 in our network. Today 99% of devices support XLAT. It’s not easy, each day is a challenge.

Subscribe to PING!

You can stream and subscribe to PING via the following channels:

• Apple Podcasts
• Google Podcasts
• Spotify
• iHeart Radio
• Pandora
• JioSaavn
• Gaana
• RSS

Rate this article

---

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.