After using several post-quantum secure VPN tunnels (here and there), I was interested in how this key exchange with ML-KEM looks on the wire. Especially in direct comparison to a classical VPN setup with just ECDH (group 21). Spoiler: since it does not fit in a single packet, it has to use a new form of fragmentation. 😳 Here we go:

Setup

This was my basic setup. A Palo Alto PA-440 with PAN-OS 12.1.6 on the left and a FortiGate FG-70F with FortiOS 7.6.7 on the right. I captured with a Network Multimeter “Allegro 500” from Allegro Packets.

I captured three runs while I changed the ciphers for both IKE and IPsec (phase 1 and phase 2) in the following way:

1. only classical elliptic curve Diffie-Hellman: group21
2. group21 + ML-KEM
3. group21 + ML-KEM + FrodoKEM + Bike (the paranoid way, just for fun)

During the VPN sessions, I only did a ping from one side to the other. Hence: Not much traffic, but only very few packets. However, note that both firewalls implement some Dead Peer Detection (DPD; ISAKMP Informational packets) methods, while the Palo additionally monitors the VPN tunnel itself (tunnel monitor; ESP packets). Hence, there are continuous packets in the trace, even without real payload traffic.

Furthermore, both ends are trying to establish the VPN at the same time. This is why you will see the “Initiator Request” and the “Responder Response” from both sides at almost the same time. But never mind. ;)

Analysis

As always, please download those merged captures to analyse them by yourself. And/or use the Ultimate PCAP, since those captures are integrated there as well. Note the packet comments to find the Start/End of the appropriate sections. (Tip: Add a custom column to display the frame.comment.)

–> PQC VPN Palo-Forti 3x different cipher-sets.pcapng.gz <–

Here’s a basic Wireshark comparison. Note the new IKEv2 messages for IKEv2 fragmentation (RFC 7383), which are a must for using IKE_INTERMEDIATE (RFC 9242) and enabling multiple key exchanges (RFC 9370).

Classical: DH group21 only: IKE_SA_INET, IKE_AUTH

Hybrid: DH group21 + ML-KEM: fragmented IKE_INTERMEDIATE

Paranoid way: DH group21 + 3x PQC: MANY fragments :D

Soli Deo Gloria!

Photo by Tobias A. Müller on Unsplash.

Since we have learned the basics of post-quantum secure VPN tunnels, let’s configure a site-to-site VPN between a Palo Alto Networks firewall and a FortiGate with PQC ciphers to verify vendor interoperability. Spoiler: it works like a charm. ;)

Palo Alto Networks supports PQC ciphers since PAN-OS 11.2, while 12.1 finally brought the ML-KEM algorithm. FortiGates support PQC ciphers since FortiOS 7.6. My lab consists of a PA-440 with PAN-OS 12.1.6 and a FG-70F with FortiOS 7.6.7.

Setup

(Note that I’m not showing a complete site-to-site VPN tutorial, but focusing on the PQC part.)

I’m using the hybrid approach. PQC ciphers are used *additionally* to the classical ECDH key agreement. That is: group21 + mlkem.

Palo Alto Networks NGFW

On the Palo side, you need to configure the IKE/IPsec Crypto, the IKE Gateway and the IPsec tunnel in the following way:

Fortinet FortiGate

On the Forti, it’s these steps that are done completely in the VPN Tunnels pane:

On the CLI, it’s this (excluding the tunnel interface and the static route). Note the set addke1 ml-kem-1024 lines.

config vpn ipsec phase1-interface
    edit "pa-lab"
        set interface "wan1"
        set ip-version 6
        set ike-version 2
        set peertype any
        set net-device disable
        set proposal aes256gcm-prfsha512
        set dhgrp 21
        set addke1 ml-kem-1024
        set nattraversal disable
        set transport udp
        set remote-gw6 2a00:6020:ad0b:8303::2
        set psksecret ENC Hr6QZodcvtxFwPEt4i1acXEZJIwGCboi1zx4+tA1CAB0zZR9jnIQMcYYpc375bFqulodhZmcN3IIA1JVaDTgK5y3gIG5VuCpkXwsaLQH8Tf4jRJEMTQYg2bBD/QpU8HGJgdEwyBvoRs880zTPoGr6fVsSLgy7smftEQi1NGzMW1M6S7QoEDyjopO6nKKQB0I/+4k6VlmMjY3dkVA
    next
end
config vpn ipsec phase2-interface
    edit "all-ipv6"
        set phase1name "pa-lab"
        set proposal aes256gcm
        set dhgrp 21
        set addke1 ml-kem-1024
        set src-addr-type subnet6
        set dst-addr-type subnet6
        set keylifeseconds 3600
    next
end

Verify

Both firewalls deliver only basic “it’s working” green icons in the GUI:

If you want to verify whether or not post-quantum ciphers are used, the CLI is a must. Note the ML-KEM keywords:

Palo:

show vpn ike-sa detail gateway <name> and

show vpn ipsec-sa tunnel <name>:

weberjoh@pa-lab> show vpn ike-sa detail gateway fg-lab 

IKE Gateway fg-lab, ID 2 2a00:6020:ad0b:8303::2 => 2a00:6020:ad0b:8303::9
  Current time: Jun.18 09:28:56

IKE SA:
  SPI:  9835E5BCE743D9A4:8153CAA28008FAA6  Init
        State:      Established
        SN:         11
        Authentication:  PSK, peer PSK
        Proposal:   AES256-GCM16/COMBINED/DH21/ML-KEM-1024/NONE/NONE/NONE/NONE/NONE/NONE
        ID local:   ipaddr:2a00:6020:ad0b:8303::2
           remote:  ipaddr:2a00:6020:ad0b:8303::9
        ID_i:       IPv6_address:2a00:6020:ad0b:8303::9
        ID_r:       IPv6_address:2a00:6020:ad0b:8303::2
        NAT:        Not detected
        Message ID: rx 6, tx 1914
        Liveness check: sending informational packet after idle 5 seconds

        Created:    Jun.18 06:49:24, 2 hours 39 minutes 33 seconds ago
        Expires:    Jun.18 14:49:24, rekey in 4 hours 8 minutes 2 seconds (24455 sec)

  Child SA 29914:
        Tunnel 2    fg-lab
        Type:       ESP       Resp
        State:      Mature
        Message ID: 00000004
        Parent SN:  11
        SPI:        A50203B8 : BEA130B7  <= ESP: BEA130B6
        Algorithm:  AES256-GCM16/COMBINED/DH21/ML-KEM-1024
        TS local:   Proto:any, ::-ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff, Ports:any
        TS remote:  Proto:any, ::-ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff, Ports:any
        Created:    Jun.18 09:08:36, 20 minutes 21 seconds ago
        Expires:    Jun.18 10:08:36, rekey in 33 minutes 2 seconds (3203 sec)



weberjoh@pa-lab> 
weberjoh@pa-lab> 
weberjoh@pa-lab> show vpn ipsec-sa tunnel fg-lab 

GwID/client IP  TnID   Peer-Address                            Tunnel(Gateway)                                                                                                                  Algorithm                                                        SPI(in)  SPI(out) life(Sec/KB)             remain-time(Sec)        
--------------  ----   ------------                            ---------------                                                                                                                  ---------                                                        -------  -------- ------------             ----------------        
2               2      2a00:6020:ad0b:8303::9                  fg-lab(fg-lab)                                                                                                                   ESP/G256/   /DH21/ML-KEM-1024                                    A50203B8 BEA130B7 3600/Unlimited           2357                     

Show IPSec SA: Total 1 tunnels found. 1 ipsec sa found.

Forti:

Please note that neither of the following two commands show the key agreement protocol, hence are not usable here: get vpn ike gateway <name>, get vpn ipsec tunnel name <name>.

–> This is the one: diagnose vpn ike gateway list name <name>:

fg-lab # diagnose vpn ike gateway list name pa-lab

vd: root/0
name: pa-lab
version: 2
interface: wan1 5
addr: 2a00:6020:ad0b:8303::9:500 -> 2a00:6020:ad0b:8303::2:500
tun_id: 10.0.0.1/::10.0.0.1
remote_location: 0.0.0.0
network-id: 0
transport: UDP
created: 78734s ago
peer-id: 2a00:6020:ad0b:8303::2
peer-id-auth: no
pending-queue: 0
PPK: no
IKE SA: created 2/5  established 2/5  time 20/4238/21070 ms
IPsec SA: created 1/25  established 1/25  time 0/866/21070 ms

  id/spi: 320 15b515dc59e0a898/cb080b494e28fca3
  direction: responder
  status: established 3212-3212s ago = 20ms
  proposal: aes256gcm
  child: yes
  SK_ei: cfbac895857d9ed8-7954388f85030dc3-f2b0e7bb050f2143-45c69edfa3d5377d-b954a6bc
  SK_er: d062865f3b430a50-795d4be04b81d0a7-5d8cea7d6665c54c-f761ddc81470c232-e64fd33e
  SK_ai: 
  SK_ar: 
  message-id sent/recv: 2/642
  QKD: no
  PQC-KEM (IKE): yes
  PQC-KEM (all IPsec): yes
  lifetime/rekey: 86400/82917
  DPD sent/recv: 00000000/00000000
  peer-id: 2a00:6020:ad0b:8303::2

That’s it. Happy networking. :)

Soli Deo Gloria!

Photo by Hugues de BUYER-MIMEURE on Unsplash.

As the advent of practical quantum computing draws closer, security vendors are increasingly introducing post-quantum cryptographic (PQC) algorithms to protect existing security architectures against future threats. One important use case is site-to-site VPN connectivity, where organisations must address the “harvest now, decrypt later” risk – the possibility that encrypted traffic captured today could be decrypted by quantum computers in the future.

To mitigate this threat, Palo Alto Networks has implemented several approaches for quantum-resistant VPN tunnels, including Post-Quantum Preshared Keys (PPK), Key Encapsulation Mechanisms (KEM), and Quantum Key Distribution (QKD). Each method offers a different balance of security, complexity, and operational requirements.

In this blog post, we will examine these approaches in detail, with a particular focus on KEM-based solutions, which are generally considered the preferred path forward. We will also demonstrate how to configure a site-to-site VPN tunnel that combines traditional Diffie-Hellman (DH) key exchange with post-quantum algorithms such as Kyber (standardised by NIST as ML-KEM), providing both classical and quantum-resistant security.

Post-Quantum Security for VPN-Tunnels

The primary challenge posed by quantum computers is their ability to break the asymmetric cryptographic algorithms that are widely used today for key exchange, most notably Diffie-Hellman (DH) and Elliptic Curve Diffie-Hellman (ECDH). As a result, we need new ways to securely establish VPN session keys over an untrusted network, such as the Internet.

Palo Alto Networks currently supports three approaches to address this challenge:

PPK (Post-Quantum Preshared Key)

Specified in RFC 8747, PPK is the simplest approach. It introduces an additional secret, known only to both VPN peers, into the key derivation process. (Do not confuse this with the traditional VPN PSK used for authentication. The PPK serves a different purpose and contributes to key establishment.)

Because the PPK is a randomly generated secret and not derived from a mathematically solvable key exchange, it remains resistant to attacks from both classical and quantum computers. Even if an attacker were able to break the DH exchange in the future, they would still require knowledge of the PPK to derive the session keys.

The downside is scalability. Every VPN tunnel requires its own securely distributed PPK, creating an additional operational burden. Furthermore, if an attacker records VPN traffic today and later obtains the corresponding PPK, the recorded traffic may become decryptable. In other words, the strong forward secrecy properties (PFS) normally provided by ephemeral key exchanges are reduced.

KEM (Key Encapsulation Mechanism)

KEM-based key exchange is expected to become the preferred solution for most deployments. Instead of relying solely on classical DH or ECDH groups, a post-quantum algorithm is used to establish shared key material. In many implementations, including current VPN solutions, the post-quantum mechanism is combined with traditional DH/ECDH in a hybrid approach, providing protection against both classical and quantum attacks.

Several post-quantum KEM algorithms have been proposed over the years. NIST has standardised CRYSTALS-Kyber as ML-KEM (Module-Lattice-Based Key Encapsulation Mechanism), which is currently the leading candidate for widespread deployment. Additional algorithms, such as HQC, may be standardised in the future as alternative options.

When using post-quantum key exchange in IKEv2, the exchange is typically performed in a hybrid manner. A classical key exchange, such as ECDH Group 21, is combined with at least one additional post-quantum key exchange round (for example, ML-KEM). The resulting key material is derived from both exchanges, ensuring that the VPN remains secure as long as at least one of the underlying mechanisms remains uncompromised.

While a single classical and a single post-quantum exchange are sufficient for most deployments, IKEv2 allows multiple additional key exchange rounds to be performed. This can further diversify the cryptographic assumptions, although the practical security benefit quickly diminishes. For most environments, one classical and one post-quantum exchange strike a sensible balance between security and complexity; configuring multiple post-quantum rounds is generally considered a rather conservative – or perhaps slightly paranoid – approach. (The German way. 😂)

QKD (Quantum Key Distribution)

This is where things become truly futuristic. :)

QKD uses the principles of quantum mechanics to exchange cryptographic key material. Any attempt to observe or intercept the quantum transmission changes its physical properties, allowing eavesdropping attempts to be detected.

In practice, QKD requires specialised hardware and a dedicated quantum communication channel, typically a fibre-optic connection between the participating sites. The generated keys are then used by conventional VPN technologies running over normal network links.

While QKD offers fascinating security properties, it also introduces high cost and operational complexity. For this reason, it is currently limited to highly specialised environments. Personally, I do not expect to deploy many QKD-enabled VPNs in the next few years – at least not in my lab. 🙂

PQ-Variants in PAN-OS Versions

Note that PANW has implemented those variants sequentially. Depending on your PAN-OS version, you’ll see none up to all three of them. ;) Links: PAN-OS 11.1, PAN-OS 11.2.

Also note that with PAN-OS 11.2 and 12.1, the list of ciphers is different:

• PAN-OS 11.2 offers various pre-standardised ciphers such as Kyber, Bike, FrodoKEM, HQC, but *not yet* the NIST standard of ML-KEM.
• PAN-OS 12.1 offers all of them as well, but additionally the NIST standard ML-KEM.

Example: S2S-VPN between 2x Palos

For this lab, I’m using 2x PA-440, one with PAN-OS 11.2.12 “pa-home”, the other with 12.1.6 “pa-lab”. Hence, I can’t use ML-KEM (since it was only added with PAN-OS 12.1), but the underlying protocol Kyber. All following screenshots/listings are from PAN-OS 12.1.6.

The first step is to add the appropriate crypto profiles for IKE as well as for IPsec: [By the way: For AES-GCM with DH group 21, sha512 is used as the PRF. This is not documented anywhere 🤦 but a logical step since group 19 uses sha256 and group 20 uses sha384, which is documented here.]

Next is to tune the IKE Gateway. Assuming you already have a working site-to-site VPN setup, you need to set the following:

• IKE Crypto Profile with PQ ciphers set a minute ago
• IKEv2 Fragmentation (otherwise you’ll get an operation failed with “pq-kem constraints failed: IKEv2 fragmentation must be enabled for PQ KEM”) with a blank line, which then defaults to some values. (There is a bug with PAN-OS 12.1.6 which sets the value to 576 if left blank. Since I’m using IPv6, I’ve set it to 1280 manually.)
• PQ KEM: Enable Post-Quantum Key Exchange and “Block IKEv2 if vulnerable cipher is used”, whatever this means.

as well as the IPsec Crypto Profile at the IPsec Tunnel:

These are the (a little weird) system logs after the tunnel establishment:

Verify!

Of course, verification is key. Search for the “KYBER-1024” entries, respectively ML-KEM, or whatever algorithm you’ve chosen.

• show vpn ike-sa detail gateway <name>
• show vpn ipsec-sa tunnel <name>

Examples:

weberjoh@pa-lab> show vpn ike-sa detail gateway pa-home

IKE Gateway pa-home, ID 1 2a00:6020:ad0b:8303::2 => 2a00:6020:ad0b:8303::1
  Current time: Jun.09 13:18:59

IKE SA:
  SPI:  CB7978FA43B5AD61:68F52ADC2342D6F1  Init
        State:      Established
        SN:         3
        Authentication:  PSK, peer PSK
        Proposal:   AES256-GCM16/COMBINED/DH21/KYBER-1024/NONE/NONE/NONE/NONE/NONE/NONE
        ID local:   ipaddr:2a00:6020:ad0b:8303::2
           remote:  ipaddr:2a00:6020:ad0b:8303::1
        ID_i:       IPv6_address:2a00:6020:ad0b:8303::2
        ID_r:       IPv6_address:2a00:6020:ad0b:8303::1
        NAT:        Not detected
        Message ID: rx 343, tx 344
        Liveness check: sending informational packet after idle 5 seconds

        Created:    Jun.09 12:50:25, 28 minutes 35 seconds ago
        Expires:    Jun.09 20:50:25, rekey in 6 hours 24 minutes 5 seconds (24760 sec)

  Child SA 347:
        Tunnel 1    pa-home
        Type:       ESP       Init
        State:      Mature
        Message ID: 0000005A
        Parent SN:  3
        SPI:        8BA354C8 : F2F00F7D
        Algorithm:  AES256-GCM16/COMBINED/DH21/KYBER-1024
        Created:    Jun.09 12:57:47, 21 minutes 13 seconds ago
        Expires:    Jun.09 13:57:47, rekey in 29 minutes 47 seconds (3060 sec)

  Child SA 600:
        Type:       INFO
        State:      Expired
        Message ID: 00000157
        Parent SN:  3




weberjoh@pa-lab>
weberjoh@pa-lab>
weberjoh@pa-lab> show vpn ipsec-sa tunnel pa-home

GwID/client IP  TnID   Peer-Address                            Tunnel(Gateway)                                                                                                                  Algorithm                                                        SPI(in)  SPI(out) life(Sec/KB)             remain-time(Sec)
--------------  ----   ------------                            ---------------                                                                                                                  ---------                                                        -------  -------- ------------             ----------------
1               1      2a00:6020:ad0b:8303::1                  pa-home(pa-home)                                                                                                                 ESP/G256/   /DH21/KYBER-1024                                     8BA354C8 F2F00F7D 3600/Unlimited           2256

Show IPSec SA: Total 1 tunnels found. 1 ipsec sa found.


weberjoh@pa-lab>
weberjoh@pa-lab>
weberjoh@pa-lab>

Soli Deo Gloria!

Photo by Dynamic Wang on Unsplash.

Fortinet has a misstep in its IPv6 settings: NAT66 is enabled by default for every policy. Not only does this make no technical sense and go against established best practices, but in my view, there’s an even bigger issue at play here:

Given the widespread use of FortiGate devices and the still limited level of IPv6 expertise among many administrators, this default setting risks creating false knowledge. Many admins may come away believing that NAT for IPv6 is just as normal as it is for IPv4 – after all, it’s enabled out of the box. And as with any default, people will quickly get used to it.

In this blog post, I’ll therefore look at a few practical workarounds to move away from this approach as quickly as possible.

Problem Description

This default configuration is fundamentally at odds with what IPv6 is meant to be – essentially the exact opposite of NAT. To recap: NAT was a workaround for IPv4, specifically to deal with address exhaustion and allow continued internet connectivity despite limited address space. The goal of IPv6, in contrast, was to restore true end-to-end connectivity at Layer 3: globally unique, routable IP addresses, with no need for NAT. (Yes, there are edge cases where NPTv6 – Network Prefix Translation – can be used appropriately.)

A single policy for IPv4 and IPv6 (“consolidated policy”) only has a single NAT switch. While this is mandatory for IPv4 to function for almost all outgoing connections, it also enables NAT66 for the same source/destination IPv6 addresses:

A forward traffic log then looks like this:

From a configuration perspective, the problem lies in the set nat enable CLI command, which does not distinguish between legacy IP and IPv6 (as the CLI *does* distinguish between both IPs when it comes to the src/dst addresses, which is not a good design as well, by the way):

config firewall policy
    edit 1
        set name "fg-lab-61 raus"
        set srcintf "fg-lab-61"
        set dstintf "wan1"
        set action accept
        set srcaddr "all"
        set dstaddr "all"
        set srcaddr6 "all"
        set dstaddr6 "all"
        set service "ALL"
        set logtraffic all
        set nat enable
    next
end

Lastly tested with a FortiGate 70F and FortiOS v7.6.6. However, this topic has been around for several FortiOS releases. I first reported it in 2017, almost 10 years ago. Unfortunately, nothing has changed since then.

Workaround 1: Central NAT

From an architect’s perspective, the central NAT is a very good option. Not only for IPv6, but also for IPv4. Why? Because it separates the network/routing layer (NAT or not) from the security layer (policy allow or deny). If you have hundreds of outgoing policies, there’s no need to have hundreds of “nat enable” switches. One NAT policy per direction fits completely, e.g.: internal -> Internet: SNAT.

Since such NAT policies are configured only for IPv4, no IPv6 NAT would be applied within the same rule. 👍🏻

Enabling Central SNAT:

Or via CLI:

config system settings
    set central-nat enable
end

Building the needed outgoing SNAT policies for legacy IP (note the IPv4 selection at the top):

Your policies will then show a “Custom” in the NAT column:

Outgoing IPv6 sessions will have no NAT anymore. ✅

Workaround 2: Independent Policies for IPv4 and IPv6

As an alternative, you can split each dual-stack policy into two policies: one for IPv4 (with NAT) and one for IPv6 (without NAT). While this will work, it obviously doubles the number of policy entries, which is, unfortunately, a really bad design as well. You’ll end up with two policy sets – one covering IPv4, the other one covering IPv6 – which won’t be congruent in your daily work. This eventually leads to more security holes in your overall policy, since you will forget one or the other.

However, technically, you would migrate from this:

to that:

Workaround 3: Different Device

I don’t know whether this is a sensible workaround or not. I also don’t want to engage in unnecessary bashing here. Still, I wonder how much you can trust a product if it already gets the most basic things completely wrong. To me, it’s more than just a slightly uneasy feeling – it’s fundamental.

Another concern is the fact that Fortinet has been aware of this issue for years, yet hasn’t made any meaningful changes. And yes, I am fully aware that FortiGate is one of the most widely deployed firewall platforms, and that this topic “only” touches the networking fundamentals rather than the security features themselves.

Even so, it raises serious questions. When customers ask me in my IPv6 workshops today which product they should choose for a new deployment, I simply cannot recommend FortiGate in good conscience. And that’s not solely because of this NAT issue – there are other IPv6-related shortcomings as well. (E.g., no address groups for v4/v6 addresses at the same time; DNS64 only globally enabled with “always-synthesize-aaaa” enabled by default; NAT64 only possible with “IP Pool” and not “outgoing interface address”; NAT46 not available with Central NAT; OSPFv3/BGP only through CLI.)

What would you do?

@Fortinet: Please fix this 🙏

Fortinet, if you’re reading this: Please reconsider this default behaviour. Please align your IPv6 implementation with the fundamental design principles of the protocol and disable NAT66 by default. This is not a minor detail – it directly shapes how administrators understand and deploy IPv6 in real-world networks. By correcting this, you would not only improve your product but also help prevent the continued spread of misconceptions around IPv6.

At the same time, I would encourage everyone reading this to raise awareness of the issue. If you are working with FortiGate devices or considering them, please reach out to Fortinet – through your account teams, support channels, or public forums – and make your voice heard. Vendor defaults matter, and change is far more likely when customers clearly and collectively highlight where improvements are needed.

Soli Deo Gloria!

Photo by Vitaly Gariev on Unsplash.

I had a use case where I wanted to use the SSL Inbound Inspection on a Palo, but with a different X.509 certificate than the one on the server itself. That is: the backend server has its self-signed (or internal PKI-signed) certificate along with its hostname, while the decryption policy on the Palo uses a publicly trusted signed certificate for the same hostname. Just like a reverse proxy / load balancer / WAF.

To be more precise about the use case: An internal server has one certificate, signed by the internal PKI, but with various DNS hostnames – internal ones, such as foobar.internal, and external ones, such as myserver.company.com.

Now, external visitors shall see a publicly trusted certificate for myserver.company.com, which is different from the one installed on the server itself. This is exactly what a reverse proxy is meant for. But to avoid another component (and since the Palo can do the security part by looking into the TLS session), I wanted to use the SSL Inbound Inspection – but with a different certificate.

By the way: Why is it still called “SSL” and not “TLS”? This really triggers me. 😂

My lab consisted of the following:

• PA-440, PAN-OS 11.2.11
• Raspi with a self-signed certificate with CN/SAN of pi10-test5.weberlab.de
• A Let’s Encrypt certificate with the same name, pi10-test5.weberlab.de
• This Let’s Encrypt certificate, including its private key, was uploaded to the Palo and used in the decryption policy.

But, as already noted, this setup is working.

The decryption log showed some “certificate verify failed” errors:

while the traffic log listed those sessions as “policy-deny”, which is not accurate from my point of view: (and those lines were not highlighted in red, though the checkbox was hit)

The show counter global revealed some ssl_inbound_cert_mismatch hits:

ssl_inbound_cert_mismatch                 70        0 error     ssl       pktproc   No matching server certificate found in config

which exactly aligns with the hit count on the decryption policy:

This KB article from PANW gives some details about that behaviour:

For whatever reason, I’m seeing a different error message. Anyway, that’s the root cause here. AFAIK, it should be possible to do this TLS interception regardless of the exact certificate being present. Hence, I suppose that it’s a limit by design.

Any comments? ;)

Soli Deo Gloria!

Photo by Robert Clark on Unsplash.

In my IPv6 classes, I always teach the basic IPv6 messages as seen on the wire. There’s so much new stuff, such as Router Advertisements, Duplicate Address Detections, Neighbour Solicitations, and so on. Therefore, I’m using a rather simple packet capture showing the starting process of a client getting its addresses.

For the last 10 years, I used this capture, which I took on a Knoppix Linux along with a German Speedport router, in which SLAAC was used for getting the addresses. However, it turned out that in enterprise-grade networks, stateful DHCPv6 is used more commonly. Hence, I did it again and captured the very first IPv6 messages as seen on an IPv6 node, but this time on a Windows 11 PC and a Debian 13, along with stateful DHCPv6.

The Packet Captures

My testbed consists of the following components:

• Windows 11 Version 25H2 (Build 26200.7462)
• Raspberry Pi 3 with Debian GNU/Linux 13 (trixie), Kernel 6.12.62+rpt-rpi-v8
• Palo Alto Networks firewall PA-440 with PAN-OS 11.1.10-h10 as the layer 3 gateway and DHCPv6 relay
• Infoblox Grid with NIOS 9.0.7, acting as the DHCPv6 server to which the Palo relays. The GUA addresses 2a00:6020:ad0b:83::/48 are out of a residential ISP connection (Deutsche Glasfaser) here in Germany. RFC 1918 for legacy IP. (Details about DHCPv6 relaying are here.)

To avoid unnecessary packets, only those clients were present on this VLAN. I captured with a real TAP, the ProfiShark, in between the switch (port A) and the clients (port B), starting *before* the LAN cables were plugged in.

I did those captures with the Windows machine (was powered up, but I plugged in and out the network cable), and then with the the Ubuntu machine, which I booted up and shut down. In both cases, I also did the following simple tasks to have some upper-layer packets in there.:

• ping -c 4 -6 weberblog.net
• ping -c 4 -4 weberblog.net
• nslookup/host securityasapodcast.de
• firefox/lynx http://ip.webernetz.net
• firefox/lynx https://ip-only.webernetz.net

Here are the capture files. Feel free to download and use them for whatever you need. (Please link back to this post if used. Thanks.) Those *.pcapng.gz files can be opened directly in Wireshark. And, of course, those packets are part of the Ultimate PCAP.

–> IPv6 Address Assignment DHCPv6 Windows.pcapng.gz <–
–> IPv6 Address Assignment DHCPv6 Linux.pcapng.gz <–

And since I was on it anyway, I did some new captures for the classical SLAAC address assignment, too. That is: RA with prefix information along with the RDNSS option:

–> IPv6 Address Assignment SLAAC Windows.pcapng.gz <–
–> IPv6 Address Assignment SLAAC Linux.pcapng.gz <–

Packets you can find within those trace files:

• Router Solicitations aka ICMPv6 type 133
• Router Advertisements aka ICMPv6 type 134 (M-flag set, no prefix; respectively prefix information and A-flag)
• Duplicate Address Detection (DAD) aka ICMPv6 type 135, sent from the unspecified address ::
• Neighbour Solicitation & Advertisement, types 135 and 136
• Multicast Listener stuff
• Stateful DHCPv6 (via relay on the Palo, only on the DHCPv6 captures)
• DNS queries
• Ping aka ICMP/ICMPv6 echo-request and echo-reply
• DHCP for legacy IP
• ARP for legacy IP

Wiresharking IPv6 –> Coloring Rules

Please note that I’m heavily relying on colouring rules in order to distinguish all those ICMPv6-based neighbour discovery protocol (NDP) messages, which are: RS/RA, NS/NA, and DAD. Furthermore, the MLD messages, which are not really of interest, are kind of hidden, since I am using a rule of green foreground with white background for them. Refer to these first four lines:

Here’s a side-by-side comparison of the *same* packets without (left) and with (right) my colouring rules:

Looking at the Startup Process

Since a picture is worth a thousand words, here are some of the bootup process from Windows with DHCPv6:

Some Notes

• Neither operating system sends Duplicate Address Detection messages for its link-local addresses, but only for their GUAs. Even though I plugged in the network cable again and again, that is, the capture process always started at the very beginning of the NIC being live on the network. Is this a current best practice I’m not aware of?
• Windows *always* sends DHCPv6 SOLICIT messages, even if the RA does not have the M-flag set, or even if no RA was received at all. Be careful in situations where both SLAAC and M-flag are used! (Not recommended anyway.)
• Quite interesting is the Internet Protocol that Windows/Linux are using for querying the recursive DNS server:
  • SLAAC with RDNSS: Windows IPv4 🟡, Linux IPv4 🟡
  • Stateful DHCPv6: Windows IPv6 ✅, Linux still IPv4 ❌ (WHY?!?!?)
  • That is: Within Dual-Stack environments, IPv6 is not used for reaching the recursive DNS server in many situations. I would have expected it to behave differently. At least for the stateful DHCPv6 part, IPv6 should be used to reach the RDNS. Maybe the stub resolver on my Linux machine behaves differently? Or is it related to latency?
• Windows does not send any Multicast Listener Reports at all. Linux does. (I don’t see any advantages of those MLDs anyway, since almost all layer 2 switches I know don’t leverage multicast snooping by default.)
• When you move a Windows PC into a new Layer 3 network, it initially assumes it is still in the previous one. You can see this as it requests the old IPv4 address via DHCP or tries to use the old IPv6 addresses via NDP. To make matters more complicated, in my two test VLANs, the Palo Alto Networks firewall (default gateway) always uses the exact same IPv6 link-local address, fe80::3efa:30ff:fe04:9114.
• I omitted many other chatty packets that were sent during the first seconds of a connected network, e.g.: NTP, LLMNR, NBNS, MDNS, IGMP, SSDP. Those are all sent for various reasons (that I don’t know) from Windows. That is: While my proposed PCAPs show all packets relevant to IPv6 and legacy IP address assignment, it does NOT show all possible name resolution and network reconnaissance attempts from Windows itself.

For the sake of completeness, here are some screenshots of how I set up the layer 3 interfaces on the Palo. (Note that I initially configured a third testbed with “stateless DHCPv6”, hence the O-flag. But since this is not a relevant scenario anymore, as all major operating systems have RDNSS implemented, I did not follow up on those captures.)

Soli Deo Gloria!

Photo by Dayne Topkin on Unsplash.

Did you know that you can easily decrypt TLS (mostly HTTPS) traffic with Wireshark? Well, only if you have the keys. ;) This really is a game-changer if you’re stuck with troubleshooting encrypted data. Let’s do an example:

The server’s private key (certificate) is no longer of interest, since all modern ciphers use (EC)DHE, in which the session keys are derived by both parties. With TLS 1.3, RSA key exchange is gone completely. All ciphers use forward secrecy.

Getting the Session Keys

In this example, I’m using a browser with HTTPS. Of course, you can decrypt any other TLS traffic with Wireshark, as long as you have the session keys. However, in most of my daily business scenarios, decrypting HTTPS is the most relevant.

You have to set the SSLKEYLOGFILE variable, which then creates a file with the session keys. No admin privileges are needed for this step. After that, open your browser of choice and start surfing. In my example, I opened my fairly simple website at https://ip.webernetz.net.

set SSLKEYLOGFILE=%USERPROFILE%\sslkeys.log
start firefox

(For some reason, various variables/fields are still named “SSL” rather than “TLS”.)

At the start of the browser, many different HTTPS sessions are already initiated by the browser itself. That is: The sslkeys.log is kind of crowded. Be aware that it contains any session keys since your start. Keep them to yourself!

In my sample, I copied only the relevant session keys for my single session. Those can be correlated by the “Random” value within the client TLS hello, field  tls.handshake.random in Wireshark. Different lines are present in the file, such as CLIENT_HANDSHAKE_TRAFFIC_SECRET, SERVER_HANDSHAKE_TRAFFIC_SECRET, CLIENT_TRAFFIC_SECRET_0, SERVER_TRAFFIC_SECRET_0, EXPORTER_SECRET. My sample keys are:

CLIENT_HANDSHAKE_TRAFFIC_SECRET 24a0ba3ed6a030ec8e3dfaaf467fedf990caa19957f92d05a2a424a8a7ec0373 afd7382d4e2250b87110fea5381e69e557d4203aab16992d4b79c2b3a54bb6a4
SERVER_HANDSHAKE_TRAFFIC_SECRET 24a0ba3ed6a030ec8e3dfaaf467fedf990caa19957f92d05a2a424a8a7ec0373 7d4920ba247e0ce9eb43bd5f2bede6e015711131fc2a943e578a8ce260f38579
CLIENT_TRAFFIC_SECRET_0 24a0ba3ed6a030ec8e3dfaaf467fedf990caa19957f92d05a2a424a8a7ec0373 68b953f99d56afffb8bcdfbe8e14fa00a7da5a3df0fee6c6ed6efd20317acf9b
SERVER_TRAFFIC_SECRET_0 24a0ba3ed6a030ec8e3dfaaf467fedf990caa19957f92d05a2a424a8a7ec0373 9833238deb0bc979f8d2384da9c318297ae89b95cd88cfed1cf6dd27ba3f3873
EXPORTER_SECRET 24a0ba3ed6a030ec8e3dfaaf467fedf990caa19957f92d05a2a424a8a7ec0373 7b95217e262cb72a82241da2785d647ea4336215f9573b9a4fc657d05e91ee84

Decryption, Please!

Now, within Wireshark, go to Edit -> Preferences -> Protocols -> TLS and select your sslkey.log file at the “(Pre)-Master-Secret log filename”:

After hitting OK, Wireshark will decrypt all applicable TLS sessions directly. The following screenshot shows the original (encrypted) PCAP on the left-hand side, while the decrypted traffic on the right-hand side. While the original TLS traffic shows only “Application Data”, which is de facto random data, the decrypted part shows the actual protocol, an HTTP GET in this scenario:

On the Packet Bytes section (bottom right), you can switch between the original Packet vs. the Decrypted TLS view:

If you want to distribute the PCAP *with* the session keys, you can inject them into the capture file, which then turns into a *.pcapng file format: Edit -> Inject TLS Secrets:

Sample in the Ultimate PCAP

The sample of HTTPS traffic shown here, including the injected TLS keys, is part of the Ultimate PCAP. Please download and analyse it by yourself – the display filter to find those packets could be: http and tcp.port eq 443.

Follow {HTTP|TCP|TLS} Stream

Note the differences between the three “Follow …” methods. The well-known “Follow TCP Stream” will still show the encrypted data:

while the “Follow TLS Stream” shows the DEcrypted data, but still with the HTTP Content-Encoding such as gzip. Hence: still not completely readable (though decrypted):

Finally, the “Follow HTTP Stream“, when clicked on an HTTP packet (rather than on a mere TCP packet), brings us readable plain-text HTTP completely:

That’s it. Happy decrypting. ;)

Further Reading/Watching

• YouTube David Bombal & Chris Greer: Decrypting TLS, HTTP/2 and QUIC with Wireshark
• “SSL/TLS decryption: uncovering secrets” presentation by Peter Wu, at SharkFest’18 EUROPE, the Wireshark Developer and User Conference
• Palo Alto Networks: Wireshark Tutorial: Decrypting HTTPS Traffic
• Netresec: How to Inspect TLS Encrypted Traffic (please note the table that lists various types of keys and configurations needed)

Soli Deo Gloria!

Photo by Markus Spiske on Unsplash.

After years of customers confusing Fortinet with Fortnite, the two companies finally decided to lean into the chaos. The result: FortiNite — a joint innovation designed to deliver “next‑gen latency acceleration” for Fortnite players worldwide, a groundbreaking collaboration with Epic Games’ Fortnite.

For years, support teams across the globe have endured phrases like “I can’t reach my Fortnite firewall” or “Is the Fortinet battle pass included?” Eventually, both companies realised:

What is FortiNite?

FortiNite is marketed as a “next‑generation, AI‑enhanced, cloud‑based latency reduction infrastructure accelerator”.

In plain English: FortiNite makes Fortnite faster. Much faster.

Leveraging Fortinet’s global Security Fabric and Epic’s galaxy of servers full of excited players, FortiNite promises:

🚀 Sub‑millisecond latency boosts
🔐 Ultra‑secure, accelerated gameplay tunnels
🎯 Skill‑based match acceleration mode
⚡ Adaptive Lag Prevention (ALP™)
🛡️ Anti‑Tilt Protection — because lag is the #1 cause of dramatic headset slams

Early internal benchmarks claim that FortiNite grants players “the equivalent of three additional shotgun reaction windows.”

Scientific? No.
Convenient? Absolutely.

Now with Full IPv6 Turbo Mode

One of FortiNite’s most celebrated features is its exclusive IPv6 Performance Pipeline.

Because while many ISPs still treat IPv6 like an optional DLC, FortiNite elevates it to its rightful place: the ultimate low‑latency gaming protocol (ULLGP).

FortiNite takes advantage of:

✨ Direct end‑to‑end IPv6 paths with fewer NAT dragons to slay
🚫 Zero NAT traversal — because NAT clearly stands for “Not Actually Tactical”
📈 Reduced congestion, thanks to IPv6 lanes that nobody else uses
💡 SmartFlow6™ — an AI engine that automatically reroutes packets away from congested IPv4 players (also known as “the lagging mortals”)

Fortinet humorously noted in their launch statement:

Epic Games added:

Integration with Existing Fortinet Products

True to the Forti‑ecosystem approach, FortiNite integrates seamlessly:

• FortiGate: Lights up in a “Victory Royale” pattern when latency dips below 5 ms
• FortiAnalyzer: Offers dashboards correlating your K/D ratio with packet‑loss spikes
• FortiManager: Lets admins distribute “No‑Build Mode” across entire VLANs
• FortiAP: Features a stealth SSID named SweatyLobby_5Ghz for peak tactical advantage

These features are, of course, “in beta” — a phrase which here means “completely fictional”.

Soli Deo Gloria!

Photo by ELLA DON on Unsplash.

You never stop learning. One topic that hadn’t crossed my path in the past decade is: Multicast. Whew. Alongside all the technical literature, online presentations, and various blog posts, I decided to approach it the classic way – through packet captures. ;)

So here’s a new part of the #UltimatePCAP, which contains quite a bit of PIM traffic, including Hello, Join/Prune, Register (via unicast!), and more. Of course, for IPv6 and legacy IP (IPv4). Let’s have a look:

Please note that I’m *not* a multicast expert. Luckily, there are many good resources out there. I recommend this Cisco Live presentation from Aleksandar Sofranic (YouTube): IP Multicast Introduction and Troubleshooting, or this (partially free) multicast course on NetworkLessons.com, or this blog post series from Emil Boklund.

Lab Setup

This is my lab, consisting of 2x Cisco router 2811 with IOS version 15.1(4)M12a and 1x Palo Alto Networks firewall PA-440 with PAN-OS 11.1.10-h1 (which only supports legacy IP for multicast, not IPv6). Everything is routed via OSPF/OSPFv3, while PIM is used for multicast traffic. The rendezvous point (RP) is statically configured on R1’s loopback address. A Raspberry Pi on the right-hand side is offering a multicast stream. Three clients are placed in the lab to receive the streams. The capture point was between the routers R1 and R2, leveraging a real TAP, namely the ProfiShark from Profitap.

Raspberry Pi ffmpeg stream (thanks to this Reddit posting):

ffmpeg -f lavfi -re -i "testsrc=size=640x360:rate=30:decimals=2" -c:v libx264 -f mpegts "udp://239.23.11.10:1234?pkt_size=1316"

Similar for IPv6:

ffmpeg -f lavfi -re -i "testsrc=size=640x360:rate=25:decimals=2" -c:v libx264 -f mpegts "udp://[ff05::2311]:1234?pkt_size=1316"

The clients/receivers used VLC to play the streams via udp://@239.23.11.10:1234, respectively udp://@[ff05::2311]:1234.

It’s Capturing Time!

This was the sequence of events that I captured for IPv4 on 2025-11-26. All times in UTC:

• 15:27 – cable between R1 and R2 was plugged into/through the TAP
• 15:31 – start of ffmpeg on the Raspberry Pi, sending to 239.23.11.10 on port 1234
  • 15:34 – client 1 starts listening to the multicast stream via VLC
  • 15:36 – client 1 stopped viewing (stream is cropped in the PCAP)
• 15:42 – stop of ffmpeg on the Raspberry Pi
  • 15:49 – client 1 starts again, though the stream is not present anymore
  • 15:50 – client 1 stops
• 15:51 – client 2 starts VLC, though the stream is not present
• 15:53 – client 2 stops

For IPv6, the following sequence was captured on 2025-12-03, all times are in UTC as well:

• 11:07 – cable between R1 and R2 was plugged into/through the TAP
• 11:10 – start of ffmpeg on Raspi, sending to [ff05::2311]:1234
  • 11:12 – client 3 starts listening to the multicast stream via VLC
  • 11:14 – client 3 stops
• 11:16 – stop of ffmpeg on the Raspi
  • 11:22 – client 3 starts again, though stream not running
  • 11:23 – client 3 stops

Some notes concerning the capture:

• I left only the beginning of the actual H.264 streams in there to keep the file as small as possible. (Can you decode them? ;))
• Neither IGMP (for IPv4) nor MLD (for IPv6) is interesting in this capture, as I captured between the two routers R1 and R2 rather than on the source or destination subnet of the multicast stream. Hence, we’re merely looking at PIM here.
• I don’t know why there’s PIMv1 traffic in there, since PIMv2 is the default at all.
• For IPv4, there are some unrelated PIM joins in there, as the clients requested some other multicast groups as well.
• Concerning IPv6, there’s a lot of ICMPv6 traffic in the capture, which relates to RS/RA, NS/NA, and MLD. I left them within the trace, as I left ARP for IPv4 there as well.

Wiresharking

Please download the UltimatePCAP by yourself in order to have a closer look at all those packets and sessions. The following screenshots give a rough overview, though.

5x for legacy IP, 5x for IPv6:

Shows & Configurations

Here are some multicast routing table outputs during the tests, all taken from R1:

As the stream started, but nobody was listening yet:

R1#show ip mroute

(*, 239.23.11.10), 00:00:25/stopped, RP 10.0.0.1, flags: SP
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list: Null

(192.168.124.12, 239.23.11.10), 00:00:25/00:02:34, flags: P
Incoming interface: FastEthernet0/1, RPF nbr 10.23.0.2
Outgoing interface list: Null

Client 1 listens to the stream:

R1#show ip mroute

(*, 239.23.11.10), 00:03:46/00:03:17, RP 10.0.0.1, flags: S
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    FastEthernet0/0, Forward/Sparse, 00:00:16/00:03:17

(192.168.124.12, 239.23.11.10), 00:03:46/00:01:13, flags: T
  Incoming interface: FastEthernet0/1, RPF nbr 10.23.0.2
  Outgoing interface list:
    FastEthernet0/0, Forward/Sparse, 00:00:16/00:03:17

Stream was not present anymore, client 2 requested it:

R1#show ip mroute

(*, 239.23.11.10), 00:00:38/00:03:27, RP 10.0.0.1, flags: S
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    FastEthernet0/0, Forward/Sparse, 00:00:38/00:03:27

Same for IPv6: Stream started, no one listening yet:

R1#show ipv6 mroute

(2A00:6020:AD0B:83C1:E37:DC1D:6FB8:B0DC, FF05::2311), 00:00:09/00:03:20, flags: SP
  Incoming interface: FastEthernet0/1
  RPF nbr: FE80::21A:6CFF:FEA1:2B98
  Outgoing interface list: Null

Client 3 listens to the stream:

R1#show ipv6 mroute

(*, FF05::2311), 00:00:37/never, RP 2A00:6020:AD0B:8399::1, flags: SCJ
  Incoming interface: Tunnel1
  RPF nbr: 2A00:6020:AD0B:8399::1
  Immediate Outgoing interface list:
    FastEthernet0/0, Forward, 00:00:37/never

(2A00:6020:AD0B:83C1:E37:DC1D:6FB8:B0DC, FF05::2311), 00:03:14/00:03:01, flags: SJT
  Incoming interface: FastEthernet0/1
  RPF nbr: FE80::21A:6CFF:FEA1:2B98
  Inherited Outgoing interface list:
    FastEthernet0/0, Forward, 00:00:37/never

Stream not present anymore, client 3 requesting it nevertheless:

R1#show ipv6 mroute

(*, FF05::2311), 00:00:02/never, RP 2A00:6020:AD0B:8399::1, flags: SCJ
  Incoming interface: Tunnel1
  RPF nbr: 2A00:6020:AD0B:8399::1
  Immediate Outgoing interface list:
    FastEthernet0/0, Forward, 00:00:02/never

For the sake of completeness, here are the configuration commands related to IP routing and multicast for both routers:

############
R1
############
ip cef
ip multicast-routing
ipv6 unicast-routing
ipv6 cef
ipv6 multicast-routing
!
interface Loopback0
 ip address 10.0.0.1 255.255.255.255
 ipv6 address 2A00:6020:AD0B:8399::1/128
!
interface FastEthernet0/0
 description Switch-gi1/0/39
 ip address 192.168.3.99 255.255.255.0
 ip pim sparse-mode
 ip ospf authentication message-digest
 ip ospf message-digest-key 1 md5 7 0331783E321E13567A053D170D175C3837
 ipv6 address 2A00:6020:AD0B:8303::99/64
 ipv6 ospf 6 area 0.0.0.0
 ipv6 ospf authentication ipsec spi 305419896 sha1 7 0327785F572B711B6B51415C35472F5A557A0D050D646D074B5F335B59750C0D0A715A563C47090A06
 bfd interval 999 min_rx 999 multiplier 3
!
interface FastEthernet0/1
 description Switch-gi1/0/40
 ip address 10.23.0.1 255.255.255.0
 ip pim sparse-mode
 ip ospf authentication message-digest
 ip ospf message-digest-key 1 md5 7 1422313E38151831102417273816502230
 ipv6 address 2A00:6020:AD0B:83C0::1/64
 ipv6 ospf 6 area 0.0.0.0
 bfd interval 999 min_rx 999 multiplier 3
!
router ospf 1
 router-id 10.0.0.1
 redistribute connected subnets
 network 10.23.0.0 0.0.0.255 area 0.0.0.0
 network 192.168.3.0 0.0.0.255 area 0.0.0.0
 bfd all-interfaces
!
ip pim rp-address 10.0.0.1
!
ipv6 pim rp-address 2A00:6020:AD0B:8399::1
ipv6 router ospf 6
 router-id 10.0.0.1
 bfd all-interfaces
 redistribute connected
!

############
R2
############
ip cef
ip multicast-routing
ipv6 unicast-routing
ipv6 cef
ipv6 multicast-routing
!
interface Loopback0
 ip address 10.0.0.2 255.255.255.255
 ipv6 address 2A00:6020:AD0B:8399::2/128
!
interface FastEthernet0/0
 description Switch-gi1/0/41
 ip address 10.23.0.2 255.255.255.0
 ip pim sparse-mode
 ip ospf authentication message-digest
 ip ospf message-digest-key 1 md5 7 133034273F1D36301F280C212F27443325
 ipv6 address 2A00:6020:AD0B:83C0::2/64
 ipv6 nd ra suppress all
 ipv6 ospf 6 area 0.0.0.0
 bfd interval 999 min_rx 999 multiplier 3
!
interface FastEthernet0/1
 description Switch-gi1/0/42
 no ip address
!
interface FastEthernet0/1.124
 description R2-Netz1
 encapsulation dot1Q 124
 ip address 192.168.124.1 255.255.255.0
 ip pim sparse-mode
 ipv6 address 2A00:6020:AD0B:83C1::1/64
!
interface FastEthernet0/1.125
 description R2-Netz2
 encapsulation dot1Q 125
 ip address 192.168.125.1 255.255.255.0
 ip pim sparse-mode
 ipv6 address 2A00:6020:AD0B:83C2::1/64
!
router ospf 1
 router-id 10.0.0.2
 redistribute connected subnets
 network 10.23.0.0 0.0.0.255 area 0.0.0.0
 bfd all-interfaces
!
ip pim rp-address 10.0.0.1
!
ipv6 pim rp-address 2A00:6020:AD0B:8399::1
ipv6 router ospf 6
 router-id 10.0.0.2
 bfd all-interfaces
 redistribute connected
!

Soli Deo Gloria!

Photo by 愚木混株 Yumu on Unsplash.

A rare use case on a Palo (at least from my point of view): Multicast Routing. And it can become as complex as you want. Fortunately, the basics are relatively easy to configure, at least if you have a rough understanding of multicast and routing with PIM and IGMP. (Recommended YouTube session here.) Let’s have a look at the common configuration steps on PAN-OS, the needed security policies to the special destination zone type of “multicast”, as well as some “show” outputs that can be used for troubleshooting:

The Lab

• My lab consists of 3x Cisco routers (2811, IOS Version 15.1(4)M12a) and 1x PA-440 “pa-lab” with PAN-OS 11.1.10-h1, ARE enabled, hence: logical routers.
• (Note that with PAN-OS 11.2 and ARE, multicast routing is not supported at all.)
• PIM-SM (sparse-mode) is used all over the lab. The rendezvous point (RP) is statically configured on R1’s loopback interface.
• Since Palo only supports legacy IP (IPv4) with multicast, IPv6 is not of interest in this setup. 😢
• A Raspberry Pi on the right-hand side offers a multicast stream at 239.23.11.10:1234. Accomplished by: ffmpeg -f lavfi -re -i "testsrc=size=640x360:rate=25:decimals=2" -c:v libx264 -f mpegts "udp://239.23.11.10:1234?pkt_size=1316".
• The Palo serves a client subnet directly (VLAN 51, client 1 aka receiver, IGMP), and is additionally connected to another “internal” router (R3, PIM), in which another receiver (client 2) resides.

The Config

The following screenshots provide an overview of the multicast settings needed for this kind of setup. Almost all settings were left at their defaults.

Special attention needs to be paid to the security policies:

• Allowing “pim” is only necessary if another router must communicate with the RP *through* the Palo, or if the Palo itself provides the RP. If the Palo is only terminating receivers (IGMP), there’s no PIM policy needed. (I’m not fully sure why, since for other routing protocols such as OSPF or BGP, those allow rules are a must. Yes, I’ve checked an explicit intrazone-deny rule as well. Still no hits.)
• IGMP must be allowed from the receiver’s zone to the special “multicast” destination zone. Of course, destination address objects can be used to further restrict the traffic.
• For the actual multicast traffic, in my case, a video stream on UDP port 1234, a policy has to allow this type of traffic from the multicast source (in my case: zone “transfer”) to the special “multicast” destination zone.

The Show

The following screenshots and CLI outputs were taken while both receivers were consuming the stream. (Using the VLC media player with udp://@239.23.11.10:1234.)

Session Browser during the stream: (the IGMP session was already gone)

Traffic Log *after* the receivers stopped consuming the stream:

Traffic Log for “pim”, in which router R3 contacted the RP:

Almost the same via the CLI show commands. One note, though: The show advanced-routing multicast pim state command is the only one that gives an output comparable to that from the Cisco world. E.g., only this one lists the Rendezvous Point for the (*,G) groups.

weberjoh@pa-lab> show advanced-routing multicast pim neighbor

Logical router: default

Interface                               Neighbor                 Uptime    Holdtime  Generation ID       DR Pri
ethernet1/1                             192.168.3.99             44:59:56  00:01:26  1170265741          1
ethernet1/5.131                         10.23.1.2                44:59:56  00:01:44  3086112133          1


weberjoh@pa-lab>
weberjoh@pa-lab>
weberjoh@pa-lab> show advanced-routing multicast igmp membership group 239.23.11.10

Logical router: default

Interface                               Address             Group               Source              Mode      Timer          Src  V    Uptime    Static
ethernet1/5.51                          192.168.51.1        239.23.11.10        *                   EXCLUDE   00:02:15       1    3    00:10:35  False
Total (Interface, Groups): 1


weberjoh@pa-lab>
weberjoh@pa-lab>
weberjoh@pa-lab> show advanced-routing multicast route group 239.23.11.10

Flags: S - Sparse, C - Connected, P - Pruned, M - SSM, R - SGRpt Pruned, F - FHR flag, T - SPT-bit set

Logical router: default

group          source              flags     Proto     incoming            outgoing            TTL  Uptime
239.23.11.10   *                   SC        IGMP      ethernet1/1         ethernet1/5.51      1    00:11:04
                                             PIM                           ethernet1/5.131     1    00:09:16
239.23.11.10   192.168.124.12      ST        IGMP      ethernet1/1         ethernet1/5.51      1    00:11:04
                                             PIM                           ethernet1/5.131     1    00:09:16
total route shown: 2


weberjoh@pa-lab>
weberjoh@pa-lab>
weberjoh@pa-lab> show advanced-routing multicast fib group 239.23.11.10

Logical Router:  default

maximum of mfib entries for this mfib:  275
number of mfib entries for this mfib:   3
number of mfib entries shown:           2

group            source           flags  incoming             outgoing
-----            ------           -----  --------             --------
239.23.11.10     0.0.0.0          1      ethernet1/1          ethernet1/5.131
                                                              ethernet1/5.51
239.23.11.10     192.168.124.12   2      ethernet1/1          ethernet1/5.131
                                                              ethernet1/5.51


weberjoh@pa-lab>
weberjoh@pa-lab>
weberjoh@pa-lab> show advanced-routing multicast pim state group 239.23.11.10

Logical router: default

(*, G):

group               RP                  up time   upstream join  join timer     RPF interface                           RPF next hop

239.23.11.10        10.0.0.1            00:11:41  Joined         00:00:20       ethernet1/1                             192.168.3.99/32

    oil interface                           local membership    join/prune          join expire timer   prune pending timer assert st           assert timer        assert winner addr  assert winner metric
    ethernet1/5.51      LOCAL               NOINFO              --:--               --:--               NOINFO              --:--               0.0.0.0             infinity

    ethernet1/5.131     NOINFO              JOIN                02:42               --:--               NOINFO              --:--               0.0.0.0             infinity

    pimreg              N/A                 JOIN                02:42               --:--               NOINFO              --:--               0.0.0.0             infinity

(S, G):

group               source              up time   upstream nbr        upstream join  join timer     RPF next hop        DR reg    DR reg stop timer   SPT

239.23.11.10        192.168.124.12      00:11:41  192.168.3.99        Joined         00:00:20       192.168.3.99/32     RegNoInfo --:--:--            False

    oil interface                           local membership    join/prune          join expire timer   prune pending timer assert st           assert timer        assert winner addr  assert winner metric
    ethernet1/5.131     NOINFO              JOIN                02:41               --:--               NOINFO              --:--               0.0.0.0             infinity


weberjoh@pa-lab>

Finally, this is how R3 looked like during the session:

R3#show ip pim neighbor
PIM Neighbor Table
Mode: B - Bidir Capable, DR - Designated Router, N - Default DR Priority,
      P - Proxy Capable, S - State Refresh Capable, G - GenID Capable
Neighbor          Interface                Uptime/Expires    Ver   DR
Address                                                            Prio/Mode
10.23.1.1         FastEthernet0/0          1d21h/00:01:39    v2    1 / G
R3#
R3#
R3#show ip mroute 239.23.11.10
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report,
       Z - Multicast Tunnel, z - MDT-data group sender,
       Y - Joined MDT-data group, y - Sending to MDT-data group,
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
 Timers: Uptime/Expires
 Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 239.23.11.10), 01:19:52/stopped, RP 10.0.0.1, flags: SJC
  Incoming interface: FastEthernet0/0, RPF nbr 10.23.1.1
  Outgoing interface list:
    FastEthernet0/1.132, Forward/Sparse, 00:10:13/00:01:56

(192.168.124.12, 239.23.11.10), 01:19:51/00:02:34, flags: JT
  Incoming interface: FastEthernet0/0, RPF nbr 10.23.1.1
  Outgoing interface list:
    FastEthernet0/1.132, Forward/Sparse, 00:10:13/00:01:56

R3#
R3#
R3#show ip igmp groups 239.23.11.10
IGMP Connected Group Membership
Group Address    Interface                Uptime    Expires   Last Reporter   Group Accounted
239.23.11.10     FastEthernet0/1.132      00:10:33  00:02:42  192.168.132.11
R3#

The End

Uff. ;) That’s it for now. Of course, you can configure other scenarios, such as setting the Rendezvous Point on the Palo itself or using filters for various settings. But not for me this time.

Soli Deo Gloria!

Photo by Jeanson Wong on Unsplash.

The other day, I was troubleshooting some network-related stuff, using the built-in Packet Capture on a Palo Alto Networks firewall. And while it did the job at a first glance, I stumbled upon some packets that were simply not correct, read: were not present on the Ethernet cable at all and/or were missing some content.

Let’s have a short look. I was configuring OSPFv3 with authentication between a Palo NGFW and a Cisco router. I wanted to validate those authentication headers (IPsec AH) with the usage of a packet capture and Wireshark. Hence, I captured on the Palo itself, filtering on the ingress interface, at all four stages. Later on, I added a real TAP, my ProfiShark from Profitap:

The interesting part is: The receive stage (rx) showed OSPFv3 packets originated and sent by the Palo firewall itself (why in the rx stage?), missing the auth header that was present in the transmit stage! That is, there are two failures in this capture: Outgoing packets are captured in the rx stage (in other words: I would think that this packet was actually *received* by the firewall), and furthermore, those packets were NOT exactly those that were present on the tx stage at all. Here’s my side-by-side comparison of the RX/TX/TAP captures:

Q.E.D.

Is this intentional? Well, concerning the “receive stage”, the docs read: “When the packet is received on the dataplane processor.” Maybe those OSPFv3 packets are forwarded/transmitted/received internally from the control plane to the dataplane, so that they appear in the receive stage? Even with an “ingress interface” set as the capture filter.

(By the way: neither the drop nor the firewall stage showed any of those OSPFv3 packets. But that’s correct to my mind, since they aren’t dropped nor passing the firewall policies.)

Well-known TAP vendors are NEOX NETWORKS or Profitap, for example.

Don’t get me wrong. Packet captures on firewalls offer a quick and easy way to get an initial look at packets. In many cases, they are fully sufficient for troubleshooting layer 7 packets that simply pass through the firewall, where you just want to inspect the contents of a DNS query, for example.

However, as soon as you’re dealing with packets that are generated or modified by the firewall itself (routing protocols, NAT, IPsec, TLS interception, etc.), you can’t fully rely on these built-in packet captures. Unfortunately, this is exactly what happens regularly – to me as well.

For further reading, please consult at least parts 4 and 5 of Jasper‘s Network Capture Playbook: SPAN Port In-Depth and TAP Basics.

Soli Deo Gloria!

Photo by Ronda Dorsey on Unsplash.

I had a hard time figuring out how to configure OSPFv3 authentication on a Palo Alto Networks NGFW due to its different configuration formats compared to a Cisco router.

TL;DR: The SPI must be set in hexadecimal, while the actual key (40 chars, hexadecimal) must be grouped in 5 sections, separated with hyphens.

Talking about Cisco’s IOS, OSPFv3 authentication is set at the interface configuration level with the following command:

ipv6 ospf authentication ipsec spi <256-4294967295> sha1 <hey-string 40 chars>

e.g.:

ipv6 ospf authentication ipsec spi 305419896 sha1 CC41D07E889B5D610FAE78E88E88C559E45D1021

However, trying to set those values on Palo’s OSPFv3 Auth Profile, I encountered the following commit error: Error: Failed to parse IPSec manual-key tunnel/profile 'ah-sha1' authentication key.

Luckily, I wasn’t the first person struggling with this, hence DuckDuckGo led me to this post: “OSPFv3 Authentication Palo Alto to Cisco Router“.

The corresponding values to the above-mentioned example are:

SPI: 0x12345678
Key: CC41D07E-889B5D61-0FAE78E8-8E88C559-E45D1021

Configured under Network -> Routing -> Routing Profiles -> OSPFv3 -> OSPFv3 Auth Profile (using a PA-440 with PAN-OS 11.2.10, Advanced Routing Engine enabled):

(Of course, other hash algorithms than SHA-1 must be used, but my lab counterparts are not capable of it. ;))

Enabled either at the OSPFv3 – Area level:

OR at the individual Interface level (preferred from my point of view):

Now, the SPI is consistent with the representation in Wireshark. Note the “Authentication Header” within the IP header, as the OSPFv3 authentication leverages the IPv6 extension header for IPsec:

Speaking about the SPI, the hexadecimal format (Palo) makes more sense compared to the decimal format (Cisco). However, I find it rather nonsensical that you have to insert hyphens in the key. 🤦‍♂️

Final note: If you have set your intrazone-default policy (default: allow) to deny, you need explicit rules for OSPF to work. Little stumbling block here: with OSPFv3 authentication, you have to allow “ipsec-ah” in addition to “ospf” in order to work. Both the session table and the traffic log will show both applications (!), dependent on the originating node, which is either the Palo or the other routers on the network: (I’ve no idea where this “from port / to port” 20033 comes from.)

Soli Deo Gloria!

Photo by George Prentzas on Unsplash.

This post guides through a basic DNS tunneling setup with the usage of the appropriate tool “iodine“. It shows how DNS tunneling works and lists the commands needed to run this type of attack. That is, you can tunnel IPv4 packets through this DNS channel via the (internal) recursive DNS resolver! Nice approach. ;)

In the end, I’m pointing out how to block these tunnelling attempts with the DNS appliances from Infoblox, and the firewalls from Palo Alto Networks and Fortinet.

At first, let’s have a look at how DNS tunneling works in general:

For more information about iodine, the DNS tunneling tool of choice, have a look at their project homepage, or this more detailed blog post from David Hamann.

iodine Setup

For my tests, I used a delegated subdomain “io.weberlab.de”. That is: underneath my domain weberlab.de, I delegated (NS records) the subdomain “io” to the IP addresses of the server, which runs iodine:

io      IN      NS      lx3
lx3     IN      A       85.215.94.29
        IN      AAAA    2a01:238:4363:ee00:9169:a8a4:e572:d5f8

Server

• -f to run in foreground
• -c for checking disabled, to answer all incoming requests
• -P passphrase
• IP address of the internal tunnel interface <- that’s the fun part
• name of the delegated zone

weberjoh@h2877111:~$ sudo iodined -f -c -P passphrase 192.168.99.1 io.weberlab.de
Opened dns0
Setting IP of dns0 to 192.168.99.1
Setting MTU of dns0 to 1130
Opened IPv4 UDP socket
Listening to dns for domain io.weberlab.de

This creates a tunnel interface called “dns0”:

weberjoh@h2877111:~$ ip a s
[...]
4: dns0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1130 qdisc pfifo_fast state UNKNOWN group default qlen 500
    link/none
    inet 192.168.99.1/27 scope global dns0
       valid_lft forever preferred_lft forever

The “vendor” of iodine offers a checking tool at https://code.kryo.se/iodine/check-it/:

Client

• -f for foreground
• -P passphrase
• -r to NOT use the raw mode which would end in a direct connection to the server rather than the usage of the recursive DNS resolver
• IP address of the recursive DNS server
• name of the delegated zone used for this tunneling (I simply used an open DNS resolver found here)

weberjoh@vm32-test2:~$ sudo iodine -f -P passphrase -r 85.214.123.36 io2.weberlab.de
Opened dns0
Opened IPv4 UDP socket
Sending DNS queries for io2.weberlab.de to 85.214.123.36
Autodetecting DNS query type (use -T to override).
Using DNS type NULL queries
Version ok, both using protocol v 0x00000502. You are user #0
Setting IP of dns0 to 192.168.99.2
Setting MTU of dns0 to 1130
Server tunnel IP is 192.168.99.1
Skipping raw mode
Using EDNS0 extension
Switching upstream to codec Base64
Server switched upstream to codec Base64
No alternative downstream codec available, using default (Raw)
Switching to lazy mode for low-latency
Server switched to lazy mode
Autoprobing max downstream fragment size... (skip with -m fragsize)
768 ok.. 1152 ok.. ...1344 not ok.. ...1248 not ok.. ...1200 not ok.. 1176 ok.. 1188 ok.. will use 1188-2=1186
Setting downstream fragment size to max 1186...
Connection setup complete, transmitting data.

weberjoh@vm32-test2:~$ ping 192.168.99.1
PING 192.168.99.1 (192.168.99.1) 56(84) bytes of data.
64 bytes from 192.168.99.1: icmp_seq=1 ttl=64 time=25.9 ms
64 bytes from 192.168.99.1: icmp_seq=2 ttl=64 time=20.0 ms
64 bytes from 192.168.99.1: icmp_seq=3 ttl=64 time=22.8 ms
64 bytes from 192.168.99.1: icmp_seq=4 ttl=64 time=19.5 ms
^C
--- 192.168.99.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 19.513/22.054/25.927/2.552 ms
weberjoh@vm32-test2:~$ lynx ip.webernetz.net

To not only ping through this DNS tunnel, but to browse, I installed the proxy “squid” on the iodine server to surf through it. Now, I was able to use this proxy (behind the internal dns0 tunnel interface on the iodine server) to browse the Internet:

weberjoh@vm32-test2:~$ export http_proxy=http://192.168.99.1:3128
weberjoh@vm32-test2:~$ lynx ip.webernetz.net

This screenshot shows the CLI-based web browser “lynx”, which I used to open https://ip.webernetz.net to show my public IP. The insight: I’m online with the IPv6 address of the iodines server rather than the (private) IPv4 address of the client itself:

Q.E.D. 😂

This Wireshark screenshot shows a capture taken on the server component of iodine during the establishment phase. (It is also part of the Ultimate PCAP. Plz download and analyse it by yourself. Display filter: dns.qry.name contains "io2.weberlab.de".) You can see normal DNS queries and responses between the DNS resolver and the iodine server itself, while the queried names are mostly random:

Blocking DNS Tunneling Attempts

Always look on the blocking side of life. 🎶

Infoblox NIOS Recursive DNS

Using Infoblox NIOS as your recursive DNS server, tunneling events such as these are blocked after just a few packets. In my tests, the tunnel was detected and blocked just after four pings (right-hand side of the screenshot):

“DNS Tunneling detected: …” events are generated, and the relevant (sub-)domain is placed in the configured blocklist RPZ:

Palo Alto Networks NGFW

Using a next-gen firewall from Palo Alto Networks between the client and the Internet, the DNS tunneling attempts with “iodine” are blocked in various ways. In fact, it wasn’t easy at all to set up a DNS tunnel through the Palo Alto firewall. 😂 I had to disable several rules and profiles just to get it working for testing purposes. It was more due to the tool’s signature than to a generic detection of DNS tunneling.

1. I had an allow rule with the application “dns”, but iodine was detected as “tcp-over-dns”, hence: blocked.
2. Then I configured a port-based (service) allow rule for TCP/UDP port 53, but still with a security group with the anti-spyware strict profile: now connections were detected as “tcp-over-dns” (allowed), but recognised as a threat “Iodine DNS Tunnel Tool Command and Control Traffic Detection” -> blocked again. :) Nice.
3. Finally, the port-based (service) allow rule without any security profile made it.

In both blocking scenarios, the DNS tunnel was already blocked during the setup period!

weberjoh@nb15-lx:~$ sudo iodine -f -P passphrase -r 85.214.123.36 io.weberlab.de
Opened dns0
Opened IPv4 UDP socket
Sending DNS queries for io.weberlab.de to 85.214.123.36
Autodetecting DNS query type (use -T to override).
Using DNS type NULL queries
Retrying version check...
Retrying version check...
Retrying version check...
Retrying version check...
Retrying version check...
iodine: couldn't connect to server (maybe other -T options will work)

Fortinet Firewall

On a FortiGate firewall, a rule allowing DNS without a security profile allowed the attack (unlike Palo Alto, which is application-based by default and blocks a “non-DNS” attempt directly).

A rule that still allows DNS, but with the Application Control to block “Proxy”, blocks iodine as well. In this blocking scenario, the DNS tunnel was already blocked during the setup period! Very good. The application was detected as “Iodine”.

weberjoh@vm32-test2:~$ sudo iodine -f -P passphrase -r 85.214.123.36 io2.weberlab.de
Opened dns0
Opened IPv4 UDP socket
Sending DNS queries for io2.weberlab.de to 85.214.123.36
Autodetecting DNS query type (use -T to override).....................
iodine: No suitable DNS query type found. Are you connected to a network?
iodine: If you expect very long roundtrip delays, use -T explicitly.
iodine: (Also, connecting to an "ancient" version of iodined won't work.)

Soli Deo Gloria!

Photo by engin akyurt on Unsplash.

On the Internet, it’s not only “always DNS” – it’s also about securing DNS. DNS faces a wide range of attack vectors, each requiring different defensive strategies. Here comes an overview of DNS security, which gives you all the keywords at a glance.

This is my approach to a picture worth a thousand words. I use it during training sessions and at customers’ sites to explain concepts related to DNS security. Of course, this drawing isn’t perfect and doesn’t show every detail, but it’s a good starting point. ;) If you have any thoughts or corrections, please leave a comment below. (Thanks to Carsten Strotmann for reviewing it.)

Feel free to use it wherever you want. Please link back to this blog post.

If you want not only the poster but also a full session that explains all of this, take a look at my DNS security presentation (full video!) from SharkFest’25 EU.

Also, have a look at my at-a-glance version for mere DNS: It’s Always DNS – Poster and its corresponding presentation.

Soli Deo Gloria!

Photo by Mark James Panaligan on Unsplash.

I was presenting at the annual “Wireshark Developer and User Conference“, the SharkFest’25 EU, talking about “Securing DNS – Attacks and Defences“. It covered all the buzzwords related to DNS security, such as malware using DNS, DNS spoofing, DNS exfiltration & tunnelling, while defending them with the keywords as DNSSEC, DoH/DoT, feeds & blocklists, and so on.

Quite many techniques. ;) Luckily, the whole session was recorded. So if you’re interested, have a look!

Here are the slides (PDF):

At the end of the talk, I showed a summary of all DNS security attack vectors and countermeasures. If you’re interested in this “simple” overview, you can find it here.

If you have any comments or questions, please go ahead and use the comment function. ;)

Me in action 😂:

Photo by Peter Conrad on Unsplash.