Seven years ago, we started an ‘IPv6 project’. The goal was to deploy IPv6 throughout our internal game studio network.
After two months of analysis, I approached my boss and recommended we kill it, at least as an ‘IPv6 project’. It was reborn as a ‘clean up our network architecture, and oh, by the way, add IPv6 (and a bunch of other things)’.
IPv6, for its own sake, offered no value to the business at all. The problems with our network were not going to be solved by adding IPv6. But there was an opportunity for a truly transformational project, a new global network architecture, designed for our business today, not the business we were 20 years ago. A new network, designed for collaboration, ease of management and service agility, which happened to include IPv6 as one of its (many) new features.
Our internal network was (and still is) complicated, as was making changes. It was difficult to respond quickly to new network and service needs. It was difficult for our users in various offices around the world to collaborate effectively. Our network, like so many other large networks, grew through osmosis over decades of rapid organic change.
About 20 years ago, what is now our internal network was three completely separate networks contained in three buildings, in three countries, with about 300 users in total. Throughout the regions (Americas, EU, and Japan) we’ve had many major acquisitions and expansions over the years, and each acquisition brought a new pre-existing network that had to be integrated as quickly as possible into its respective regional network. In 2005, we began to inter-connect (but not integrate) the three regional networks.
Today, our internal networks support 14 game development studios around the world. Almost 4,000 people with at least 15,000 devices use the networks every day to create award-winning video games for PlayStation game consoles. (Our consumer-facing networks support all the many thousands of people who play our games each day, but that’s a story for a future post.)
IPv6 allows us to simplify our network
The problem with our network was not that it didn’t have IPv6. The problem was that it was the result of the merger of dozens of networks over a 20-year period. When the original pieces of our networks were first conceived (in the early 1990s), the first PlayStation console was still in the future, cell phones were the size of bricks, online gaming was in its infancy, and there was (practically) no world wide web.
I didn’t see this network until the early 2000s, and it was already the case that our network was the most complex network many of us had ever worked on. Our backgrounds ranged from ISPs, to government agencies, to supercomputing, to large research universities.
By around 2011, we began redesigning the entire global network from scratch. Just getting to a new design was not an overnight process. We had to start with current and future needs, learn about IPv6, create technical guiding principles, create global governance, and even mundane things like IP address plans.
IPv6 is a key technology that will help us simplify, re-factor and reimagine our worldwide network. Too much of the complexity of our network was due to the vision of IPv4 addresses as a precious commodity to be hoarded, and to the avoidance of network renumbering during acquisitions. We joked that from some parts of our network to another, a single packet might be NATed, re-NATed, over-NATed, under-NATed, de-NATed, and then NATed again. Not quite true, but an indication of the complexity of our inter-connected networks.
The network we are (still) building is a completely new design. We started with an empty whiteboard, network and technology architects from all our global regions, and a list of all the things that we felt was wrong with our network, and all things that were right. We designed a new network that is focused on the way our business works today, that enables easy collaboration between our development studios, and makes it easier to bring new technologies to our users.
The new network will be simpler, with much of the legacy and IPv4 complexity left behind. In concept, it is an IPv6 network, that just happens to still support IPv4 for a while. We tried to leave behind all the IPv4-thinking, such as address scarcity, the need for slicing subnets ever smaller into /26, /30, NAT, and private IP space (RFC1918). There are no transition technology cul-de-sacs.
We still have legacy IPv4, and RFC1918 space, and all our old networks, but our network team has been ripping out NATs, legacy firewall rules and other accumulated cruft.
Our EU network team worked with the rest of the IT organization to completely renumber thousands of devices, including desktops, PlayStation devices, printers and such to the new IPv4/IPv6 address plan, in support of the network simplification.
The new network will be fully dual-stacked for now, and frankly, for the foreseeable future. All IPv6 addresses are Globally Unique Addresses, and our ‘site’ subnets are /48s so that they can be individually announced via BGP to all our network providers when or if necessary.
We’re rolling out the new network when we have time and when needed. We’re building the new network a few subnets at a time, then slowly moving services, users, their devices and desktops onto the new network. All our client devices are already dual-stack capable, and we are dual-stacking servers and services when we move them too. When the old IPv4-only subnets are empty, they are just decommissioned.
Author’s note: In 2016, we merged two large companies into a single company, to which we added another company in 2017. All the complexity I’ve described was before that merger even started. We’re now working towards the merger of three separate company networks, which I plan to write about in a future post.
Original post appeared on Tom Perrine’s Blog, Thuktun.
Tom Perrine is a long-time information security practitioner and system administrator, who is currently an Enterprise Architect for a global video game company.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.
It would be interesting if a follow-up blog came out putting economic numbers and ROI behind all the technology description.
Thanks for sharing your practical experience with IPv4 / IPv6, transition and dual-stack. Allow me to share a bit of our experience in terms of address space management.
A few years ago, we accidentally ventured into studying the IPv4 address pool exhaustion challenge, perhaps due to the curiosity from our telephony background. We now have submitted a proposal, called EzIP (phonetic for Easy IPv4) to IETF:
https://tools.ietf.org/html/draft-chen-ati-adaptive-ipv4-address-space-03
EzIP will not only resolve IPv4 address shortage issues, but also largely mitigate cyber security vulnerabilities, plus open up new possibilities for the Internet. These should relieve the urgency to move onto the IPv6. Originally, our efforts were inspired by two regularly updated worldwide statistics:
https://ams-ix.net/technical/statistics/sflow-stats/ether-type
https://stats.labs.apnic.net/ipv6
So, we thought that initial EzIP targets would be emerging regions and rural areas of developed countries where assignable IPv4 addresses are in short supply. A recent article about the Internet activities provided a surprising new perspective:
https://dyn.com/blog/ipv6-adoption-still-lags-in-federal-agencies/
It concluded that the IPv6 adoption even at US Federal Agencies was moving at “a glacial pace”. This seems to imply that the entire market for alternatives to the IPv6 approach, such as the EzIP, is now open. The general public should be equally informed of this kind of choices, instead of being led by the existing industrial interests. Hopefully, these provide you some updated references.
Feedback and comments are very much appreciated.
Abe (2018-06-24 12:38)
Dear Colleagues:
0) Here are two pieces of updated information for share:
1) The following is a discussion thread on the “state of IPv6”. The findings are quite surprising.
http://www.circleid.com/posts/20190529_digging_into_ipv6_traffic_to_google_is_28_percent_deployment_limit/
2) Then, you may like to have a look at the feasibility demonstration report below about our proposed architecture for expanding IPv4 address pool, addressing ITU’s CIR proposal, etc.:
https://www.avinta.com/phoenix-1/home/RegionalAreaNetworkArchitecture.pdf
These should provide some material for furthering the dialog
Abe (2020-08-30 16:34 EDT)