Deploying IPv6 at IBM

How do you embrace IPv6 in a global enterprise, with tens of thousands of network devices, delivery teams all around the globe, and no real shortage of IPv4 space?

In this article, I will describe how we introduced IPv6 inside IBM’s own network over the past few years, and look at the challenges we encountered as well as what lies on the road ahead. As enterprises, in general, seem to lag behind in adopting IPv6 as compared to ISPs or carriers, this article is aimed at helping those who still need to build a case within their own company.

Across the Internet ecosystem, people are busy doing things with IPv6. Mobile carriers have it, broadband carriers do it, content providers offer it, operating systems use it, the Cloud wants it, and IoT needs it. But all too often, and especially when we look at enterprises, what’s keeping the people pushing for IPv6 busiest, is convincing key stakeholders that they should invest in their own deployment.

Seven steps towards your rollout

1. Convince your decision makers

When conversations about IPv6 start taking place in an enterprise, it’s vital that you make the case and have your message resonate with decision makers. One of the ways we did this at IBM was to point to product lifetimes.

When you tell your finance team that a product you’re about to install, which works fine without IPv6 right now, will inevitably need to be ripped out long before its expiry date once IPv6 becomes a necessity, you’re not only showing them how they can influence rollout costs through timing, you’re also giving them a good reason to get started fast — ideally now.

2. Nominate a technical champion and find an executive sponsor

You should nominate a technical IPv6 champion. Ideally, this is a person with a technical mindset and who is passionate about IPv6. He or she will be the one to start technical conversations with other senior architects in your organization. This will help them understand what IPv6 means for their respective area.

At a certain size, this technical role can’t be done as a ‘side-job’, in addition to a busy daily schedule. So either allow this person sufficient time or better still, dedicate a person to this role. Remember, your promise to finance was that by doing this, you will save money in the long run. This might help fund the additional headcount needed.

If you have an established structure of ACBs (architecture control boards), ensure that IPv6 is added as something that should always be asked about during the review process.

In addition to the technical role, nominate one person from your executive team to be the ‘IPv6 sponsor’. That person doesn’t necessarily need to be an IPv6 expert. However, having one at this level will help to have questions about IPv6 raised again and again when new projects are launched, funding plans created and so forth.

3. Get seed funding

You should aim to get some seed funding for IPv6. You don’t need to secure a multi-million dollar budget at the start. You can start small and grow over time. But don’t wait!

Once you have some funding, I would advise not wasting a lot of your time looking at legacy services. If you focus on new services, as we did in our case, you might be able to dovetail on their development and leverage their rollout effort.

Try to certify every new service being introduced as being IPv6 ready. Also, the willingness of these teams to look at IPv6 is likely to be much higher than of those managing an established service or application. Overall, this focus on new services is likely to yield quicker results than trying to fight difficult cases.

Once you get a better understanding of your detailed IPv6 architecture and the features you need, start updating your purchasing guidelines for equipment, so that your company only procures devices that will offer the features you need in the future. To start with, you could read this document: ripe-554 — we used an approach analogous to it. The key is to identify the particular features you will need at different levels of your network hierarchy. You won’t need everything in every device! And don’t trust the vendor if all they have to show is a single green check-mark saying ‘IPv6-ready’ 😉

4. Be an evangelist!

In my opinion, and this is possibly underestimated by the technically-minded, be an evangelist!

As important as it is to have the technical conversations with the tech team, it’s no less important to reach out to as many groups as possible in your organization. Talk to the hosting staff, the front-end load-balancer team, your client OS folks, building management, endpoint management and security. Basically, talk to as many groups as you can. Keep talking about IPv6 and spread the word. If nothing else, at this early stage of the game, it could be as little as getting your company more prepared for IPv6.

5. Find role models

Whether you’re a corporate enterprise or a multinational, it’s good to go out and look at examples of organizations like yours, who’ve already gone towards IPv6. That said, many of the examples you’ll come across will be targeted towards providers and carriers. But there are others out there who’ve done it: talk to your peers, other enterprises, and corporations of the same size, in the same area or industry.

Attend conferences, where you can meet like-minded people. Some of them may not have shared their approach in public, but are willing to engage in a private exchange of knowledge between the technical teams of their company and yours. You can learn from their achievements and their mistakes, though of course, in the end, you need to make your own decision. There is no right or wrong.

6. Make an IPv6 addressing plan

This is a big one. Once you have made the decision to move to IPv6, you need to develop an IPv6 addressing plan. Setting up a good addressing plan that fits your organization, the structure and topology of your network and your future plans, is key.

One important item for us, as a multinational, was to decide where to get our IPv6 allocation from. Do you just use one prefix from one of the Regional Internet Registries (RIRs) or do you approach each of them and get separate prefixes per region? There’s no single answer to this question, as the best approach depends on the specifics of your network. In our case, we decided to get just one prefix for our internal network. However, internally, we split the allocation into parts dedicated to specific regions so that, in the worst case, we would only have to renumber a region and not the entire network.

Then, once you have received the IPv6 prefix allocated to your organization, you need to think about the structure of your network and what hierarchy you want to build in, your backbone, and your current network topology (which is likely going to change). You also need to look at how much you want to base it on, say, geography or on the proximity of sites (which might be a more stable parameter).

Although you may have a pretty good idea of what your network and business might look like in five years, it’s usually difficult to go much beyond that. Therefore you need to come up with a plan that will remain stable enough to last, but which is also flexible enough that you can accommodate the unexpected.

To summarize, I can only recommend that you take your time to make a proper addressing plan early on.

7. Go native!

We are still heavily dependent on IPv4 these days, so the only large-scale way to deploy IPv6 right now, for us, is to go with dual-stack. But this can only be a transition architecture because it means we’re running two IP stacks in parallel, which requires extra maintenance.

I recommend that you deploy native IPv6 wherever you can so as to avoid tunnelling. The beauty of dual-stack is that you can still use your existing network management tooling. Your endpoints will choose the best way to connect — over IPv4 or IPv6. This option gives people some comfort that there is still IPv4 to rely upon.

However, the additional burden of two stacks is something to consider in the long run. We want to get away from it eventually and get back to maintaining just a single stack in our network: IPv6. We all know that IPv4 will probably be around for decades in some parts of the network. But I recommend building it then as a service encapsulated on top of IPv6, rather than running it natively on your backbone forever.

Spreading IPv6 around the globe

Once you’ve taken these initial steps, it’s time to roll out. We operate about 40,000 routers and switches and another 20,000 access points for our offices around the globe. We have very stringent standards of how we roll things out at IBM sites and people sometimes ask me if there is a dedicated IPv6 team. No, there isn’t. You basically need to pull together the experts from each network discipline: the campus architect, the wireless architect, the data centre architect. They really need to understand what IPv6 means in their specific environment.

After we made architectural decisions, we tested and certified the solutions in our lab before starting to roll them out to a few pilot sites. When these were successful, we applied these tested configurations into production with the regular delivery teams in a larger environment.

The positive side-effect of identifying sites in every economy around the globe means that you then get a much larger team involved, rather than just your senior architects. All these delivery and service management teams now need to familiarize themselves with IPv6. There is no longer an excuse because it has become a ‘normal’ part of the standard configurations.

They will, in return, very likely come back to you with questions or findings from the operational aspect, with things you may not have thought of when developing the architecture. So it will start some very good discussions.

With the thorough testing we had done beforehand, we were able to make this IPv6 rollout a very smooth one and had very few production issues. You know it’s a compliment, and a testament to your technical teams when your global operational lead (the person usually involved in troubleshooting and fire-fighting problems) tells you: I really haven’t heard much about IPv6.

Achievements

The things we’ve achieved to support IPv6 in the past few years are:

• We have native wide-area transport for traditional MPLS or sites over Internet VPN.
• We offer IPv6 for LAN and Wireless LAN, for end users and into the data centre.
• We offer IP services such as DNS, DHCP and route it outbound on Internet egress gateways.
• We updated our asset inventory so that we could do IPv6 rollout tracking.

So far, we have covered more than 100 sites around the globe, the footprint of over 100,000 IBM staff. The majority of them are actively using IPv6.

As soon as you offer IPv6 to clients or endpoints, they will start picking it up and you will see IPv6 production traffic in your network. The majority of our production traffic at this point is destined to IPv6-capable websites and services on the Internet. In parallel, we are starting to enable internal services to also be offered over IPv6.

Looking at the AP region, we started our deployment there after the first successful pilots and deployments done in Europe and North America.

Initially, we struggled to get traction in the region, because we were lacking someone to manage the processes for us locally, on the ground (within timezone and culture). Things started to really take off when we found a very capable, dedicated project manager in India, who has helped us overcome this barrier.

Abhishek wasn’t an IPv6 expert when he joined, but really this is not something you require to run this kind of project. More important was that he learnt very quickly and managed the technical teams efficiently. From a project management perspective, this is a network/infrastructure project roll-out, not so much different from other network projects. As such, Abhishek’s expertise has grown from deploying IPv6 for ‘simple’ end user sites into very complex data centre environments.

Meanwhile, we have expanded into many different economies in the region, including Indonesia, South Korea, Malaysia, Viet Nam and Taiwan — there is no economy left where we have not done at least a few IBM sites.

Overall, it has become business-as-usual to leverage existing project lifecycles, such as an equipment replacement, to drive IPv6 along with it as a standard service.

This was the whole premise in the beginning: to embed IPv6 into the regular project lifecycle, rather than funding a dedicated rollout project.

Technical difficulties on the way

Bugs

It turned out to be difficult to properly plan the amount of time needed to develop a solution, because of a number of bugs we ran into. You need to test, test, and test again. During our lab tests, we came across numerous bugs and worked with our vendors to have them fixed (before we rolled out).

One of the things we learned is that vendors treat bug reports very differently. Some of them are quite responsive and provide solutions, others say “It’s a feature, not a bug”. The quality of resolution depends a lot on how seriously the vendor treats IPv6 in their products.

Monitoring

I said above that dual-stack is nice because you can fall back on IPv4, but it also has a downside because it can lead to not noticing when something is wrong with IPv6. The end users might not even notice IPv6 failing because IPv4 is still working. Even if everything works fine in the beginning, you really need to have a good monitoring system in place, so you can detect any problems with either of the protocol stacks.

Lack of support for enterprises

One of the big issues I noticed when deploying IPv6 at an enterprise is some lack of standards support from the rest of the industry. A lot of transition technologies and other IETF work are geared primarily towards carriers and ISPs, and are not necessarily applicable to enterprises.

Multihoming

If you want to do something like hybrid WAN, where you have some larger consolidated egress gateways in a region, but you also want to do a local breakout in your sales offices, you probably need to think about some form of a translation. Just using a firewall routed egress with preserving the source address is not possible.

In IPv4, you use policy-based routing and both the local breakout and the central part will have different IPv4 NAT pools. So, by nature, the return path will be symmetrical and you’ll come back in through the same firewall as you left. That’s not the case if you want to preserve the IPv6 source address.

There is ongoing work to see how one can use multiple source addresses based on the service you want to reach. But it is still under discussion. It’s not something you will find in products today.

Remember to share your journey

In summary, although enterprises are often still bringing up the rear when it comes to IPv6 deployment, there are plenty of reasons to take the lead, and plenty of routes we can use to get there. Be an evangelist, speak up, go to as many people as you can, and do internal briefings. Also, as you do begin to make headway, remember to be a role model. Tell other technical communities how you got there and make them aware of the benefits IPv6 can bring to them in their environment.

This would not have been possible without many helping hands. For their special contribution across the AP region, I’d like to thank Abhishek Parmar and Ana Espinosa; it’s a pleasure to run this program with you!

Adapted from original post on RIPE Labs Blog.

Andy Mindnich is the Global IPv6 and IP services architect for IBM’s internal network. He is also responsible for DNS, DHCP and IPAM services inside IBM globally.

Rate this article

---

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.