IPv6 renumbering: a pain in the …

By on 20 Sep 2018

Category: Tech matters

Tags: , , ,

4 Comments

Blog home

Photo: 'Safety in Numbers' by Craig Rodway, Flickr

If you follow my blog, you probably know that I am using IPv6 everywhere. Everything in my lab is dual-stacked if not already IPv6-only.

It’s been great so far, however, a few months ago, my lab moved to another ISP, which required me to change all IP addresses (since I don’t have provider-independent (PI) space yet). [Insert expletive].

While it was almost no problem to change the legacy IPv4 addresses (only a few NATs), it was a huge pain in the … to change the complete infrastructure with its global unicast IPv6 addresses. It turned out that changing the interface IPv6 addresses was just the first step, while many modifications at different services were the actual problem. And this was only my lab and not a complete company or the like.

The following is a list of changes I did for IPv6 and for legacy IP — just an overview to get an idea of differences and stumbling blocks.


Key Points:

  • Moving to another ISP required changing all IP addresses.
  • Changing public IPv4 addresses was done in a few minutes. Changing public IPv6 addresses took me a couple of weeks.
  • It was not only changing the interface IPv6 addresses but also lots of configuration details in almost all services/appliances where IPv6 GUAs were used as well.

However, please note that this is NOT a problem of IPv6 in general but due to the fact that we can use ‘public’ IPv6 address (GUAs) everwhere. This is only needed when you’re forced to change your IPv6 prefix which you should avoid completely! If you would use public IPv4 addresses all over your network (as it was intended a couple of decades ago), you would have the same problems for IPv4 too. It’s the NAT for IPv4 that saves us from renumbering our networks when changing the ISP. I won’t start the “don’t use ULAs” discussion here for now. 😉 (Reminder: Use PI space!)

Before you begin, you might want to look at RFC 5887 – Renumbering Still Needs Work. While most of this RFC focuses on the renumbering of interface addresses, at least section 5.3.4 – Management Issues discusses a few scenarios in which IPv6 addresses are used in other configuration files that pose problems. (And of course: changing an IPv6 prefix for a client-only network using SLAAC or stateful DHCPv6 is no problem at all — probably even with cascaded routers using DHCPv6-PD — but this post is focused on servers.)

Understanding my lab

My lab consists of many different hardware devices and virtual appliances along with open source software (see below).

A (probably not complete) list of my lab
Palo Alto Networks firewall, Fortinet FortiGate firewall, Dell switch, Cisco routers and switches, Pulse Connect Secure remote access VPN, F5 BIG-IP load balancer, Cisco ESA Email Security Appliance, Perle console server, VMware ESXi Server, couple of Raspberry Pis, Ubuntu server VMs, BIND name servers (authoritative as well as recursive caching), NTP servers, syslog-ng server, RIPE Atlas probe, Postfix mail transfer agent, MRTG monitoring server, ntopng real-time network analyzer, Apache web server.

 

My prefix changed from 2003:51:6012::/48 to 2003:de:2016::/48. I absolutely love my new prefix because it is so easy to remember. Oh yeah. The first hextet ‘2003’ is quite common, the second hextet ‘de’ is just as the TLD for Germany, and the third hextet ‘2016’ is just like the year 2016. Lovely. ♥♥♥

However, since the third hextet changed from ‘6012’ to ‘2016’, some typos were inevitable. Since most (but not all) systems were dual-stacked, I almost always used an IPv4 admin connection to change the IPv6 stuff.

One positive side effect during this renumbering was that I actually enforced my host ID structure. Formerly I was quite lazy and simply increased the rightmost bits beginning with ::1, whereas I used some kind of logic within my new IPv6 prefix.

Changes overview

The following table summarizes the number of changes I did for both Internet protocols. In fact, it is a list that compares the usage of IPv6 GUA addresses vs IPv4 private addresses in server configurations.

IPv6Legacy IP
Host/Device/VM Interface Addresses23x (address + gateway + DNS server)None! LOL Everything with private addresses
Firewall: Interfaces7x2x
Firewall: Static Routes10x2x
Firewall: Host/Network Objects29x9x
Firewall: NATsNOT A SINGLE ONE! STRIKE! This is why we're all here. 😉1x outgoing, 0x incoming since objects (row above) were used
Firewall: VPNs (RA & S2S)3x10x (since most VPNs are over v4)
Firewall: Miscellaneous6x (custom reports, LLDP profile, DNS-Proxy, RDNSS)None
DNS Names AAAA and A47x23x
DNS Authoritative Zones: masters, also-notify5x4x
DNS Glue Record: ns1.weberdns.de1x1x
Reverse DNS PTR Records 40x (incl. new zone for new v6 range)None (since internal RFC 1918 addresses haven't changed)
Syslog Forwarding11xNone
NTP Restricts (for access from MRTG)6xNone
SNMP read access2x1x
Pulse Connect Secure: Admin Auth Policy1x1x
Pulse RA VPN: Client Addressing1xNone
ntopng: local-networks1xNone
DNS caching BIND: allow-recursion1xNone
RIPE Atlas measurements2x1x
Postfix: relayhost, mynetworks2xNone
Cisco ESA: Host Access Table (HAT), Relaylist 4xNone
MRTG: Targets w/ static IPs3xNone
FileZilla FTP Server: NAT external IPNone1x

Note, the number of ‘None’ counts for IPv4, while for IPv6 I had to adjust several configuration files. Counting the mere numbers, it’s 205x changes of IPv6 addresses vs 56x changes of legacy IPv4 addresses. It’s not just the numbers that count, but the dependencies.

Some details

While the mere changes of interface IPv6 addresses of all devices were quite obvious, here are some examples where those addresses were used in other configurations that did not work until I corrected them to my new IPv6 range.

Some of them came immediately into my mind, while some others did not work for weeks until I encountered some smaller errors.

Postfix

Straightforward: Replacing the old IPv6 addresses for the relayhost and mynetworks. Easy:

relayhost = [2003:de:2016:110::d01b:25]
mynetworks = 127.0.0.0/8 [::1]/128 192.168.0.0/16 [2003:de:2016:100::]/56

I tried using an FQDN for the relayhost but it used only the IPv4 address of the DNS record. Since I wanted to force v6 transmission, I was stuck with using the raw IPv6 address.

Cisco Email Security Appliance

Another example that required configuration changes was my email routing on the Cisco Email Security Appliance (ESA).

Outgoing mails through the RELAYLIST as well as SMTP routes used the old IPv6 addresses. At least for the SMTP routes, I was able to use FQDNs rather than IP(v6) addresses, while the tested FQDNs for the HAT didn’t work:


Figure 1 — HAT with old IPv6 addresses.


Figure 2 — SMTP routes with new FQDNs.


Figure 3 — SMTP routes with old IPv6 addresses.

BIND caching DNS Server

Uh, this was a hard one. I am using a BIND server as my caching DNS server. Of course, I am not allowing anyone to query it, but only my own IP ranges.

Now here is the point: I totally forgot to adjust the ‘allow-recursion’ statement AND I did not recognize it for a couple of weeks. Why? Because only some monitoring servers are using this server and I was not looking at them at the time. Just after I saw some errors on one of those servers, I troubleshot the issue and finally found the error on the BIND server. Of course, in a user environment, this would have been reported within minutes. This is the corrected line:

allow-recursion { localhost; 192.168.0.0/16; 2003:de:2016:100::/56; };

syslog-ng

Though there was no problem sending syslog messages to the new address, all logs were stored in newly created folders since I am using the source IP addresses for the folder generation. That is, searching for logfiles from certain devices now implies using the old and the new hierarchy of folders. Bad.

This listing shows part of my syslog-ng root folder. The first lines are from the old prefix; new prefix start in the middle:

weberjoh@jw-nb10-syslog-mirror:/var/log/firewalls$ ls -ahl
total 340K
drwxr-xr-x 85 root root   4.0K Apr 19 08:07 .
drwxrwxr-x 17 root syslog 4.0K Apr 27 06:46 ..
drwxr-xr-x  3 root root   4.0K Aug 29  2017 ::1
drwxr-xr-x  3 root root   4.0K May 30  2017 2003:51:6012:110::19
drwxr-xr-x  3 root root   4.0K Jun 16  2017 2003:51:6012:110:213:95ff:fe24:3404
drwxr-xr-x  3 root root   4.0K May 30  2017 2003:51:6012:110::6b5:123
drwxr-xr-x  3 root root   4.0K May 30  2017 2003:51:6012:110::a07:53
drwxr-xr-x  3 root root   4.0K May 30  2017 2003:51:6012:110::b12:22
drwxr-xr-x  3 root root   4.0K May 30  2017 2003:51:6012:110::b15:22
drwxr-xr-x  3 root root   4.0K May 30  2017 2003:51:6012:110::dcf7:123
drwxr-xr-x  3 root root   4.0K Jun 20  2017 2003:51:6012:110::dcfb:123
drwxr-xr-x  3 root root   4.0K May 30  2017 2003:51:6012:120::13
drwxr-xr-x  3 root root   4.0K May  9  2017 2003:51:6012:120::2
drwxr-xr-x  3 root root   4.0K May  9  2017 2003:51:6012:120::24
drwxr-xr-x  3 root root   4.0K May  9  2017 2003:51:6012:120::25
drwxr-xr-x  3 root root   4.0K May 30  2017 2003:51:6012:120::a08:53
drwxr-xr-x  3 root root   4.0K May 30  2017 2003:51:6012:120::a16:53
drwxr-xr-x  4 root root   4.0K Aug 29  2017 2003:de:2016:110::6b5:123
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:110::a07:53
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:110::a12:443
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:110::b12:22
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:110::b15:22
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:110::dcf7:123
drwxr-xr-x  4 root root   4.0K Jan  6 07:05 2003:de:2016:110::dcfb:123
drwxr-xr-x  3 root root   4.0K Jan 12 09:16 2003:de:2016:110:f264:1d36:a4a3:8ded
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:120::a01:80
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:120::a08:53
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:120::a16:53
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:120::d05:22
drwxr-xr-x  3 root root   4.0K Apr  6 12:59 2003:de:2016:120::e51a:22
drwxr-xr-x  3 root root   4.0K Apr  6 13:48 2003:de:2016:120::e51b:22
drwxr-xr-x  4 root root   4.0K Jan  1 00:00 2003:de:2016:120::f01:443

Palo Alto firewall services

Just two examples from my Palo Alto Networks firewall in which IPv6 GUAs are used as well:

  1. Email Server Profile (for outgoing alert emails through the Cisco ESA appliance) and,
  2. The RDNSS option within the router advertisements for network interfaces.


Figure 4 — Palo Alto Networks Email Server Profile.


Figure 5 — Palo Alto Networks RDNSS.

RIPE Atlas measurements

I am using a couple of RIPE Atlas measurements, including for my authoritative DNS servers. Since the IP addresses (in this case: IPv6 and IPv4) have changed for one of those servers, some areas turned red and never recovered — it ended up starting new measurements resolving to the new addresses.


Figure 6 — Changing both IPv4 and IPv6 addresses caused an error for my RIPE Atlas measurments.

Conclusion

Long story short: Avoid renumbering! Go for PI space! This is the common advice [1, 2, 3].

As I showed, there are many situations where we’re using IPv6 addresses in configuration files that end up with interrupted services after renumbering. Some of them will remain faulty for a long period (such as my allow-recursion for BIND) since it’s almost impossible to prove that all your services are fully migrated to the new prefix.

And this post was only about my lab and not about a historically grown company.

Citing RFC 5887 again:

“It should be noted that the management and administration issues (i.e., tracking down, recording, and updating all instances where addresses are stored rather than looked up dynamically) form the dominant concern of managers considering the renumbering problem. This has led many sites to continue the pre-CIDR (Classless Inter-Domain Routing) approach of using a provider-independent (PI) prefix.”

 

One final hint: Use FQDNs rather than raw addresses when possible!

This is not possible everywhere, for example, in almost all of my config files IP(v6) addresses are mandatory since the config file is NOT able to do a DNS query before it gets read out by the service itself. At least for some cases such as my MRTG monitoring server sending a SNMP get message, the use of FQDNs rather than raw IPv6 addresses is a better solution.

Adapted from original post, which appeared on blog.webernetz.net

Johannes Weber is a network security consultant at Webernetz.net.

Rate this article

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.

4 Comments

  1. Mark Smith

    RFC5887,

    “As an additional result of the multi-addressing mechanism, a site
    might choose to use Unique Local Addressing (ULA) [RFC4193] for all
    on-site communication, or at least for all communication with on-site
    servers, while using globally routeable IPv6 addresses for all off-
    site communications.”

    “This would make these on-site activities immune to
    renumbering of the prefix(es) used for off-site communication.”

    Reply
  2. Lawrence Hughes

    Forward into the past! NAT is NOT a good thing. We broke the IPv4 Internet in the mid 90’s with it, splintering the Internet into millions of private internets. Now we are all second class netizens, only able to make outgoing connections from out tiny fiefdoms. IPv6 restores the monolithic address space and allows any node to connect to any node again – full decentralization. Don’t throw that away because you don’t have good tools. Our DNS appliance can do instant prefix renumbering. We are working on a next gen IPAMS that really understands IPv6. We are working on messaging apps that can go directly from my phone to yours – no intermediate server. IPv4 is DEAD. Bury it, and NAT along with it. See http://www.thirdinternet.com.

    Reply
  3. Etuate Cocker

    I wouldnt ignore the fact that NAT is still useful today. Agree that we need to migrate to IPv6 but from an enterprise network perspective the cost of transition and the need to build specialised tools (that you mentioned) are some of reasons why ISPs and customers are reluctant to move from V4. Perhaps we should discuss it from a customer’s perspective instead of the Engineering mindset?

    Reply
    1. Mark Smith

      There are plenty of problems caused by NAT, and since it’s primary purpose was to overcome lack of IPv4 addresses, there isnt a need for it in IPv6.

      APNIC asked me to write a series blog articles on the trouble with NAT a few years ago, which is based on experience with NAT starting in 1996. I also discuss the IPv6 alternatives in part 3.

      https://blog.apnic.net/author/mark-smith/

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Please answer the math question * Time limit is exhausted. Please reload CAPTCHA.

Top