During my IPv6 classes, I discovered a (minor) bug at the NGFW from Palo Alto Networks: ICMPv6 error messages, such as “time exceeded” (type 3) as a reply of traceroute, or “destination unreachable” (type 1) as a reply of a drop policy, are not correctly sourced from the IPv6 address of the data interface itself, but from the unspecified address “::”. Here are some details:
Tested environments: PA-220 with PAN-OS 10.2.4-h2 and 10.2.5, as well as PA-440 with PAN-OS 11.0.2-h2. [UPDATE] This bug (PAN-214336) is now fixed with PAN-OS 10.2.8, 11.0.3, and 11.1.0.[/UPDATE]
The Bug
I basically stumbled upon this problem as I played around with some traceroutes:
|
1 2 3 4 5 6 7 8 9 10 |
weberjoh@nb15-lx:~$ traceroute -6 lx2.weberlab.de traceroute to lx2.weberlab.de (2001:470:1f0b:16b0::a36:22), 30 hops max, 80 byte packets 1 :: (::) 1.095 ms 0.763 ms 0.755 ms 2 * * * 3 * * * 4 * * * 5 port-channel12.core2.fra1.he.net (2001:470:0:63d::1) 3.155 ms 3.590 ms * 6 tserv1.fra1.he.net (2001:470:0:69::2) 46.521 ms 49.557 ms 52.595 ms 7 tunnel525322-pt.tunnel.tserv6.fra1.ipv6.he.net (2001:470:1f0a:16b0::2) 124.070 ms 122.339 ms 121.901 ms 8 2001:470:1f0b:16b0::a36:22 (2001:470:1f0b:16b0::a36:22) 80.968 ms 77.495 ms 74.469 ms |
In Wireshark, this looks as follows:
Later on, I discovered the same behaviour when a security policy’s action is set to “Drop” along with “Send ICMP Unreachable”, e.g. here:
This looks as follows in Wireshark (different ICMPv6 type, but the same wrong address):
RFC 4443 “Internet Control Message Protocol (ICMPv6) for the Internet Protocol Version 6 (IPv6) Specification” defines how the source address of an originating ICMPv6 packet MUST look like:
The very only case, in which an IPv6 packet is sent from the unspecified address, is the Duplicate Address Detection process and its corresponding Neighbour Solicitation. (RFC 4861 and RFC 4291.)
Two more notes:
- This only happens with IPv6. For legacy IP, the correct data interface IPv4 address is chosen as the source address.
- This only appears on VLAN interfaces, but not on “normal” Layer 3 interfaces. <- This is interesting because some years ago I found another IPv6 bug which was related to a VLAN interface as well. To me, this seems like the IPv6 routines for different types of interfaces on PAN-OS are not exactly the same.
On a mere Layer 3 interfaces those ICMPv6 packets are sourced correctly:
PAN Ticket -> Bug
In October 2022, I created a ticket at the PAN support, telling them about this bug. Luckily, I was able to convince the ticket owner that this has to be fixed since it is clearly a violation of the Internet standards. (To be fair, I also told them that I don’t believe that this is a big thing nor security-related.) The bug got the bug ID PAN-214336. After some months, I got an update from the engineering team, saying that this should be fixed with PAN-OS 10.2.5. Unfortunately, this wasn’t the case. Neither is this bug listed in the release notes nor is it actually fixed with PAN-OS 10.2.5, as I tested it again. Furthermore, it is also present in the PAN-OS 11.x major versions.
Hopefully, it will get fixed someday. At least, you guys can google it now. :)
Fixed with PAN-OS 10.2.8, 11.0.3, 11.1.0
Finally, in February 2024, this bug is fixed. I was able to test this with a PA-220 and PAN-OS 10.2.8 (addressed issues in the release notes), as well as with a PA-440 with PAN-OS 11.0.3 (nothing in the release notes). I was told that it was also fixed in PAN-OS 10.1.12 and 11.1.0, but I have not tested it by myself.
Soli Deo Gloria.
Photo by Valeriy Borzov on Unsplash.
Copied/pasted from my comments on X/Twitter on the request of the blog author. I will not be monitoring replies here though.
Twitter/X context:
https://x.com/DaryllSwer/status/1706617274939896013?s=20
Comment 1:
If I read your blog post correctly, you receive packets with :: as source address from interfaces that do not have layer 3 reachability right i.e. no GUA address and I assume it’s inter-VLAN routing?
In which case :: is the correct source address selection:
https://www.rfc-editor.org/rfc/rfc4291.html#section-2.5.2
Comment 2:
Similar case here, of how :: traverse the DFZ. As AS9198 explained in the comments and as per the RFC, :: is used when an interface has no address:
https://labs.ripe.net/author/kistel/the-curious-case-of-packets-from/
Hey Daryll. Thanks for your comment. It is inter-VLAN routing, but *with* layer 3 addresses (LL and GUA) on the VLAN interfaces. Hence, addresses are in place but not used.
And: wow, thanks for the interesting read about “::” on RIPE Atlas. Even though this 6PE is an explanation, you’re still right to my mind that we should not see those packets in the Internet at all.
Thanks again!