IPv6 renumbering: a pain in the …

If you follow my blog, you probably know that I am using IPv6 everywhere. Everything in my lab is dual-stacked if not already IPv6-only.

It’s been great so far, however, a few months ago, my lab moved to another ISP, which required me to change all IP addresses (since I don’t have provider-independent (PI) space yet). [Insert expletive].

While it was almost no problem to change the legacy IPv4 addresses (only a few NATs), it was a huge pain in the … to change the complete infrastructure with its global unicast IPv6 addresses. It turned out that changing the interface IPv6 addresses was just the first step, while many modifications at different services were the actual problem. And this was only my lab and not a complete company or the like.

The following is a list of changes I did for IPv6 and for legacy IP — just an overview to get an idea of differences and stumbling blocks.

Key Points:

• Moving to another ISP required changing all IP addresses.
• Changing public IPv4 addresses was done in a few minutes. Changing public IPv6 addresses took me a couple of weeks.
• It was not only changing the interface IPv6 addresses but also lots of configuration details in almost all services/appliances where IPv6 GUAs were used as well.

However, please note that this is NOT a problem of IPv6 in general but due to the fact that we can use ‘public’ IPv6 address (GUAs) everwhere. This is only needed when you’re forced to change your IPv6 prefix which you should avoid completely! If you would use public IPv4 addresses all over your network (as it was intended a couple of decades ago), you would have the same problems for IPv4 too. It’s the NAT for IPv4 that saves us from renumbering our networks when changing the ISP. I won’t start the “don’t use ULAs” discussion here for now. 😉 (Reminder: Use PI space!)

Before you begin, you might want to look at RFC 5887 – Renumbering Still Needs Work. While most of this RFC focuses on the renumbering of interface addresses, at least section 5.3.4 – Management Issues discusses a few scenarios in which IPv6 addresses are used in other configuration files that pose problems. (And of course: changing an IPv6 prefix for a client-only network using SLAAC or stateful DHCPv6 is no problem at all — probably even with cascaded routers using DHCPv6-PD — but this post is focused on servers.)

Understanding my lab

My lab consists of many different hardware devices and virtual appliances along with open source software (see below).

My prefix changed from 2003:51:6012::/48 to 2003:de:2016::/48. I absolutely love my new prefix because it is so easy to remember. Oh yeah. The first hextet ‘2003’ is quite common, the second hextet ‘de’ is just as the TLD for Germany, and the third hextet ‘2016’ is just like the year 2016. Lovely. ♥♥♥

However, since the third hextet changed from ‘6012’ to ‘2016’, some typos were inevitable. Since most (but not all) systems were dual-stacked, I almost always used an IPv4 admin connection to change the IPv6 stuff.

One positive side effect during this renumbering was that I actually enforced my host ID structure. Formerly I was quite lazy and simply increased the rightmost bits beginning with ::1, whereas I used some kind of logic within my new IPv6 prefix.

Changes overview

The following table summarizes the number of changes I did for both Internet protocols. In fact, it is a list that compares the usage of IPv6 GUA addresses vs IPv4 private addresses in server configurations.

Note, the number of ‘None’ counts for IPv4, while for IPv6 I had to adjust several configuration files. Counting the mere numbers, it’s 205x changes of IPv6 addresses vs 56x changes of legacy IPv4 addresses. It’s not just the numbers that count, but the dependencies.

Some details

While the mere changes of interface IPv6 addresses of all devices were quite obvious, here are some examples where those addresses were used in other configurations that did not work until I corrected them to my new IPv6 range.

Some of them came immediately into my mind, while some others did not work for weeks until I encountered some smaller errors.

Postfix

Straightforward: Replacing the old IPv6 addresses for the relayhost and mynetworks. Easy:

relayhost = [2003:de:2016:110::d01b:25]
mynetworks = 127.0.0.0/8 [::1]/128 192.168.0.0/16 [2003:de:2016:100::]/56

I tried using an FQDN for the relayhost but it used only the IPv4 address of the DNS record. Since I wanted to force v6 transmission, I was stuck with using the raw IPv6 address.

Cisco Email Security Appliance

Another example that required configuration changes was my email routing on the Cisco Email Security Appliance (ESA).

Outgoing mails through the RELAYLIST as well as SMTP routes used the old IPv6 addresses. At least for the SMTP routes, I was able to use FQDNs rather than IP(v6) addresses, while the tested FQDNs for the HAT didn’t work:

Figure 1 — HAT with old IPv6 addresses.

Figure 2 — SMTP routes with new FQDNs.

Figure 3 — SMTP routes with old IPv6 addresses.

BIND caching DNS Server

Uh, this was a hard one. I am using a BIND server as my caching DNS server. Of course, I am not allowing anyone to query it, but only my own IP ranges.

Now here is the point: I totally forgot to adjust the ‘allow-recursion’ statement AND I did not recognize it for a couple of weeks. Why? Because only some monitoring servers are using this server and I was not looking at them at the time. Just after I saw some errors on one of those servers, I troubleshot the issue and finally found the error on the BIND server. Of course, in a user environment, this would have been reported within minutes. This is the corrected line:

allow-recursion { localhost; 192.168.0.0/16; 2003:de:2016:100::/56; };

syslog-ng

Though there was no problem sending syslog messages to the new address, all logs were stored in newly created folders since I am using the source IP addresses for the folder generation. That is, searching for logfiles from certain devices now implies using the old and the new hierarchy of folders. Bad.

This listing shows part of my syslog-ng root folder. The first lines are from the old prefix; new prefix start in the middle:

weberjoh@jw-nb10-syslog-mirror:/var/log/firewalls$ ls -ahl
total 340K
drwxr-xr-x 85 root root   4.0K Apr 19 08:07 .
drwxrwxr-x 17 root syslog 4.0K Apr 27 06:46 ..
drwxr-xr-x  3 root root   4.0K Aug 29  2017 ::1
drwxr-xr-x  3 root root   4.0K May 30  2017 2003:51:6012:110::19
drwxr-xr-x  3 root root   4.0K Jun 16  2017 2003:51:6012:110:213:95ff:fe24:3404
drwxr-xr-x  3 root root   4.0K May 30  2017 2003:51:6012:110::6b5:123
drwxr-xr-x  3 root root   4.0K May 30  2017 2003:51:6012:110::a07:53
drwxr-xr-x  3 root root   4.0K May 30  2017 2003:51:6012:110::b12:22
drwxr-xr-x  3 root root   4.0K May 30  2017 2003:51:6012:110::b15:22
drwxr-xr-x  3 root root   4.0K May 30  2017 2003:51:6012:110::dcf7:123
drwxr-xr-x  3 root root   4.0K Jun 20  2017 2003:51:6012:110::dcfb:123
drwxr-xr-x  3 root root   4.0K May 30  2017 2003:51:6012:120::13
drwxr-xr-x  3 root root   4.0K May  9  2017 2003:51:6012:120::2
drwxr-xr-x  3 root root   4.0K May  9  2017 2003:51:6012:120::24
drwxr-xr-x  3 root root   4.0K May  9  2017 2003:51:6012:120::25
drwxr-xr-x  3 root root   4.0K May 30  2017 2003:51:6012:120::a08:53
drwxr-xr-x  3 root root   4.0K May 30  2017 2003:51:6012:120::a16:53
drwxr-xr-x  4 root root   4.0K Aug 29  2017 2003:de:2016:110::6b5:123
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:110::a07:53
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:110::a12:443
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:110::b12:22
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:110::b15:22
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:110::dcf7:123
drwxr-xr-x  4 root root   4.0K Jan  6 07:05 2003:de:2016:110::dcfb:123
drwxr-xr-x  3 root root   4.0K Jan 12 09:16 2003:de:2016:110:f264:1d36:a4a3:8ded
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:120::a01:80
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:120::a08:53
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:120::a16:53
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:120::d05:22
drwxr-xr-x  3 root root   4.0K Apr  6 12:59 2003:de:2016:120::e51a:22
drwxr-xr-x  3 root root   4.0K Apr  6 13:48 2003:de:2016:120::e51b:22
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:120::f01:443

Palo Alto firewall services

Just two examples from my Palo Alto Networks firewall in which IPv6 GUAs are used as well:

1. Email Server Profile (for outgoing alert emails through the Cisco ESA appliance) and,
2. The RDNSS option within the router advertisements for network interfaces.

Figure 4 — Palo Alto Networks Email Server Profile.

Figure 5 — Palo Alto Networks RDNSS.

RIPE Atlas measurements

I am using a couple of RIPE Atlas measurements, including for my authoritative DNS servers. Since the IP addresses (in this case: IPv6 and IPv4) have changed for one of those servers, some areas turned red and never recovered — it ended up starting new measurements resolving to the new addresses.

Figure 6 — Changing both IPv4 and IPv6 addresses caused an error for my RIPE Atlas measurments.

Conclusion

Long story short: Avoid renumbering! Go for PI space! This is the common advice [1, 2, 3].

As I showed, there are many situations where we’re using IPv6 addresses in configuration files that end up with interrupted services after renumbering. Some of them will remain faulty for a long period (such as my allow-recursion for BIND) since it’s almost impossible to prove that all your services are fully migrated to the new prefix.

And this post was only about my lab and not about a historically grown company.

Citing RFC 5887 again:

“It should be noted that the management and administration issues (i.e., tracking down, recording, and updating all instances where addresses are stored rather than looked up dynamically) form the dominant concern of managers considering the renumbering problem. This has led many sites to continue the pre-CIDR (Classless Inter-Domain Routing) approach of using a provider-independent (PI) prefix.”

One final hint: Use FQDNs rather than raw addresses when possible!

This is not possible everywhere, for example, in almost all of my config files IP(v6) addresses are mandatory since the config file is NOT able to do a DNS query before it gets read out by the service itself. At least for some cases such as my MRTG monitoring server sending a SNMP get message, the use of FQDNs rather than raw IPv6 addresses is a better solution.

Adapted from original post, which appeared on blog.webernetz.net

Johannes Weber is a network security consultant at Webernetz.net.

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.