IPv6 Renumbering: A Pain in the …

If you’re following my blog you probably know that I am using IPv6 everywhere. Everything in my lab is dual-stacked if not already IPv6-only. Great so far.

A few months ago my lab moved to another ISP which required to change all IP addresses (since I don’t have PI space yet). Oh boy! While it was almost no problem to change the legacy IPv4 addresses (only a few NATs), it was a huge pain in the … to change the complete infrastructure with its global unicast IPv6 addresses. It turned out that changing the interface IPv6 addresses was merely the first step, while many modifications at different services were the actual problem. And this was *only* my lab and not a complex company or the like.

Following you find a list of changes I made for IPv6 and for legacy IP. Just an overview to get an idea of differences and stumbling blocks.

TL;DR: Changing the public IPv4 addresses was done in a few minutes, while it took me a couple of weeks until all services were running with the new IPv6 addresses likewise. It was not only changing interface IPv6 addresses, but lots of configuration details in almost all services/appliances where IPv6 GUAs were used as well.

However, please note that this is NOT a problem of IPv6 in general, but due to the fact that we can use “public” IPv6 address (GUAs) everwhere. This crap here is only needed when you’re forced to change your IPv6 prefix which you should avoid completely! If you would use public IPv4 addresses all over your network (as it was intended a couple of decades ago), you would have the same problems for IPv4, too. It’s the NAT for IPv4 that saves us from renumbering our networks when changing the ISP. I won’t start the “don’t use ULAs” discussion here for now. ;) Just the reminder: Use PI space!

You might want to look at RFC 5887 “Renumbering Still Needs Work”. While most of this RFC focuses on the renumbering of interface addresses, at least section 5.3.4 “Management Issues” discusses a few scenarios in which IPv6 addresses are used in other configuration files that pose problems. (And of course: changing an IPv6 prefix for a client-only network using SLAAC or stateful DHCPv6 is no problem at all. Probably even with cascaded routers using DHCPv6-PD. But this post is focused on servers.)

Understanding my Lab

My lab consists of many different hardware devices and virtual appliances along with open source software. A (probably not complete) list of it is: Palo Alto Networks firewall, Fortinet FortiGate firewall, Dell switch, Cisco routers and switches, Pulse Connect Secure remote access VPN, F5 BIG-IP load balancer, Cisco ESA Email Security Appliance, Perle console server, VMware ESXi Server, couple of Raspberry Pis, Ubuntu server VMs, BIND name servers (authoritative as well as recursive caching), NTP servers, syslog-ng server, RIPE Atlas probe, Postfix mail transfer agent, MRTG monitoring server, ntopng real-time network analyzer, Apache web server.

My prefix changed from 2003:51:6012::/48 to 2003:de:2016::/48. I absolutely love my new prefix since it is that easy to remember. Oh yeah. The first hextet “2003” is quite common, the second hextet “de” is just as the TLD for Germany, and the third hextet “2016” is just like the year 2016. Lovely. ♥♥♥ However, since the third hextet changed from “6012” to “2016”, some typos were inevitable. Since most (but not all) systems were dual-stacked, I almost always used an IPv4 admin connection to change the IPv6 stuff.

One positive side effect during this renumbering was that I actually enforced my host ID structure. Formerly I was quite lazy and simply increased the rightmost bits beginning with ::1, whereas I used some kind of logic within my new IPv6 prefix. (Blogpost following.)

Changes Overview

The following table summarizes the number of changes I did for both Internet protocols. In fact it is a list that compares the usage of IPv6 GUA addresses vs. IPv4 private addresses in server configurations. Some more details below.

IPv6Legacy IP
Host/Device/VM
Interface Addresses
23x (address + gateway + DNS server)none! LOL. Everything with private addresses
Firewall: Interfaces7x2x
Firewall: Static Routes10x2x
Firewall: Host/Network Objects29x9x
Firewall: NATsNOT A SINGLE ONE! STRIKE! This is why we're all here. ;)1x outgoing, 0x incoming since objects (row above) were used
Firewall: VPNs (RA & S2S)3x10x (since most VPNs are over v4)
Firewall: Miscellaneous6x (custom reports, LLDP profile, DNS-Proxy, RDNSS)none
DNS Names AAAA and A47x23x
DNS Authoritative Zones: masters, also-notify5x4x
DNS Glue Record: ns1.weberdns.de1x1x
Reverse DNS PTR Records40x (incl. new zone for new v6 range)none (since internal RFC 1918 addresses haven't changed)
Syslog Forwarding11xnone
NTP Restricts (for access from MRTG)6xnone
SNMP read access2x1x
Pulse Connect Secure: Admin Auth Policy1x1x
Pulse RA VPN: Client Addressing1xnone
ntopng: local-networks1xnone
DNS caching BIND: allow-recursion1xnone
RIPE Atlas measurements2x1x
Postfix: relayhost, mynetworks2xnone
Cisco ESA: Host Access Table (HAT), Relaylist4xnone
MRTG: Targets w/ static IPs3xnone
FileZilla FTP Server: NAT external IPnone1x

Note the “none”s for IPv4, while for IPv6 I had to adjust several configuration files. Counting the mere numbers it’s 205x changes of IPv6 addresses vs. 56x changes of legacy IPv4 addresses. But it’s not just the numbers that count, but the dependencies:

Some Details

While the mere changes of interface IPv6 addresses of all devices was quite obvious, here are some examples where those addresses were used in other configurations that did not work until I corrected them to my new IPv6 range. Some of them came immediately into my mind, while some others did not work for weeks until I encountered some smaller errors.

Postfix

Straight forward: Replacing the old IPv6 addresses for the relayhost and mynetworks. Easy:

I tried using an FQDN for the relayhost, but it used only the v4 address of the DNS record. Since I wanted to force v6 transmission, I was stuck using the raw v6 address.

Cisco ESA Email Security Appliance

Another example which required configuration changes was my email routing on the Cisco ESA. Outgoing mails through the RELAYLIST as well as SMTP routes used the old v6 addresses. At least for the SMTP routes I was able to use FQDNs rather than IP(v6) addresses, while the tested FQDNs for the HAT didn’t work:

BIND caching DNS Server

Uh, that was a hard one. I am using a BIND server as my caching DNS server. Of course I am not allowing anyone to query it, but only my own IP ranges. Now here is the point: I totally forgot to adjust the “allow-recursion” statement AND I did not recognize it for a couple of weeks. :D Why? Because only some monitoring servers are using this server and I was not looking at them at the time. Just after I saw some errors on one of those servers, I troubleshoot the issue and finally found the error on the BIND server. Of course, in an user environment, this would have been reported within minutes. ;) This is the corrected line:

syslog-ng

Though there was no problem sending syslog messages to the new address, all logs were stored in newly created folders since I am using the source IP addresses for the folder generation. That is: searching for logfiles from certain devices now implies using the old and the new hierarchy of folders. Bad.

This listing shows my syslog-ng root folder. Line 16-34 are from the old prefix; starting with line 35 the new prefix kicks in:

 

Palo Alto Firewall Services

Just two examples from my Palo Alto Networks firewall in which IPv6 GUAs are used as well: email server profile (for outgoing alert emails through the Cisco ESA appliance) and the RDNSS option within the router advertisements for network interfaces:

RIPE Atlas Measurements

I am using a couple of RIPE Atlas measurements, e.g. for my authoritative DNS servers. Since the IP addresses (in this case: v6 and v4) have changed for one of those servers, some areas turned red and did never recover. It ended up starting new measurements resolving to the new addresses.

Conclusion

Long story short: Avoid renumbering! Go for PI (provider-independent) space! This is the common advise: reference1 (Ivan), reference2 (Enno), reference3 (Greg). As I showed, there are many situations where we’re using IPv6 addresses in configuration files that end up with interrupted services after renumbering. Some of them will remain faulty for a long period (such as my allow-recursion for BIND) since it’s almost impossible to prove that all your services are fully migrated to the new prefix. And this post was only about my lab and not about a historically grown company.

Scheiße 97/366” by Dennis Skley is licensed under CC BY-ND 2.0

Citing RFC 5887 again: “It should be noted that the management and administration issues (i.e., tracking down, recording, and updating all instances where addresses are stored rather than looked up dynamically) form the dominant concern of managers considering the renumbering problem. This has led many sites to continue the pre-CIDR (Classless Inter- Domain Routing) approach of using a provider-independent (PI) prefix.”

One hint at the end: Use FQDNs rather than raw addresses! This is not possible everywhere, for example, in almost all of my config files IP(v6) addresses are mandatory since the config file is NOT able to do a DNS query before it gets read out by the service itself. At least for some cases such as my MRTG monitoring server sending SNMP get message, the usage of FQDNs rather than raw IPv6 addresses is the better solution.

Cheers. ;)

Note that this post is one of many related to IPv6. Click here for a structured list.

Featured image “Safety In Numbers” by Craig Rodway is licensed under CC BY-NC-ND 2.0.

10 thoughts on “IPv6 Renumbering: A Pain in the …

  1. Hmm, in some recent post or article you mentioned also the risks of VPN traffic escaping unencrypted when using GUAs. Somehow I have the impression you recommend global IPv6 addresses throughout an enterprise network?
    Personally, I would use ULAs where private IPv4 is used. This creates a clear separation between internal and DMZ systems, that everybody in the organization would understand.
    NAT is another issue, but proxies, loadbalancers or ALGs should always be preferred over NAT, which sometimes creates more issues then it actually resolves in an enterprise network IMHO.

    1. Hey Peter.

      And here we go again: The ULA vs. GUA discussion again. ;)

      Yes, (among almost any other IPv6 related persons out there) I highly recommend the usage of GUAs throughout the enterprise network. This is the huge advantage of IPv6 that you need NOT to use any translation such as NAT or proxies to access the destination. ULAs and NPTv6 or even NAT66 bring many new problems. Please don’t stuck with this “IPv4-thinking” of “private addresses are good”. They aren’t. Neither from an operational perspective, nor from a security perspective. Refer to: https://weberblog.net/why-nat-has-nothing-to-do-with-security/

      Concerning my post about the VPN routing: Yes, you must be careful when routing your networks into VPN tunnels. Even more when it comes to IPv6 compared to IPv4. But this is not an argument of any kind to use ULAs instead of GUAs.

      (There is this small exception for using ULAs: When you’re a very small enterprise without PI-space, you should use ULAs with NPTv6 to be not stucked with one ISP. But only then. Refer to: https://blog.ipspace.net/2014/01/pa-pi-or-ula-ipv6-address-space-it.html)

      Concerning “This creates a clear separation between internal and DMZ systems”: I don’t think that we need this clear separation. What for? DMZs are now the cloud, while “internal” is now “home office workers”. Hence there is no clear separation anyway. ;)

      Thanks for reaching me out. Please post your comment.

      Ciao,
      Johannes

      1. And why not using both ULA and GUA ?

        ULA for everythinng that’s yours, every networking inside so you don’t renumber everything like that happens and GUA for external traffic. Every device gets a GUA and a ULA. It even protects devices that should not be reached out by the internet (for example, why would your syslog servers listen to a public internet adress ?)

        No NAT at all but just the intended way ULA should work.

        1. sounds good. ULA only for internal only applications and GUA for The Internet and beyond ;D .
          This is also a great advantage of IPv6: Multiple addresses per Interface. This makes multi WAN and internal-only addressing more straight forward.

  2. I’m curious for your IPv6-only network(s) are you using any type of v4 translation such as NAT64? If so what implementation are you using?

  3. Arguably, you are just doing it all wrong? I mean, sure, you might have to do some more manual changes than with an RFC1918 network, but I think quite a bit of the work should be avoidable.

    In particular, IMO, it’s best practice nowadays to make the logical structure of IT services independent of the network structure that connects them as much as possible. Consider it a violation of the abstraction when you configure IP addresses at the application layer. For example, you just shouldn’t do email relay access control based on IP addresses: Use TLS, and either cert-based auth or SMTP AUTH to grant relay access, and suddenly, changes of addresses are completele irrelevant. Also, that’s not only useful for renumbering, it’s just as useful for moving machines: If you want to move some machine to be hosted elsewhere, say, it can relay emails just as before.

    And if you really need to do address-based access control, IMO, it’s best to do it on your firewall, and set up your firewall in such a way that you can just swap out the prefix in one place and be done. Or rather, support multiple simultaneous prefixes, so you can add a new prefix, age out the old one, and then remove it. Services should, if possible, just bind to in6addr_any, and allow access from anything that the firewall lets through.

    Similarly, you should just auto-generate addresses in your zone files, so you only need to swap out the prefix in one place to point clients to the new addresses.

    If you build your network this way, renumbering for many services is just:

    1. add new prefix in firewall

    2. add new prefix in RAs

    3. change prefix in zone files

    4. deprecate old prefix in RAs

    5. remove old prefix from firewall

    All with apropriate delays in between, of course, to let old DNS records and old prefixes age out.

    And really, IMO, there is no reason to not also configure servers via SLAAC, including virtual machines/containers. The reasons against DHCP for servers don’t really apply: The address is deterministic without any state that is held by the RAD, and thanks to unsolicited RA addresses get assigned as soon as the RAD becomes available, not only after some DHCP client retry timeout.

    Also, IMO, software that considers the resoluton of host names a startup-time task is buggy, and that becomes particularly obvious with IPv6: For one, resolution failures for host names in the config should cause the software to just disable any functionality that is dependent on knowing the address and keep retrying, re-enabling the functionality when successful. But also, software should respect the TTLs in DNS records and re-resolve names when the TTL expires, and reconfigure when the resulting addresses change. That would solve a lot of renumbering problems by concentrating all the necessary changes to the name server, where it further could be concentrated to a single place where you specify the prefixes.

Leave a Reply

Your email address will not be published. Required fields are marked *