Did you know that you can easily decrypt TLS (mostly HTTPS) traffic with Wireshark? Well, only if you have the keys. ;) This really is a game-changer if you’re stuck with troubleshooting encrypted data. Let’s do an example:

The server’s private key (certificate) is no longer of interest, since all modern ciphers use (EC)DHE, in which the session keys are derived by both parties. With TLS 1.3, RSA key exchange is gone completely. All ciphers use forward secrecy.

Getting the Session Keys

In this example, I’m using a browser with HTTPS. Of course, you can decrypt any other TLS traffic with Wireshark, as long as you have the session keys. However, in most of my daily business scenarios, decrypting HTTPS is the most relevant.

You have to set the SSLKEYLOGFILE variable, which then creates a file with the session keys. No admin privileges are needed for this step. After that, open your browser of choice and start surfing. In my example, I opened my fairly simple website at https://ip.webernetz.net.

set SSLKEYLOGFILE=%USERPROFILE%\sslkeys.log
start firefox

(For some reason, various variables/fields are still named “SSL” rather than “TLS”.)

At the start of the browser, many different HTTPS sessions are already initiated by the browser itself. That is: The sslkeys.log is kind of crowded. Be aware that it contains any session keys since your start. Keep them to yourself!

In my sample, I copied only the relevant session keys for my single session. Those can be correlated by the “Random” value within the client TLS hello, field  tls.handshake.random in Wireshark. Different lines are present in the file, such as CLIENT_HANDSHAKE_TRAFFIC_SECRET, SERVER_HANDSHAKE_TRAFFIC_SECRET, CLIENT_TRAFFIC_SECRET_0, SERVER_TRAFFIC_SECRET_0, EXPORTER_SECRET. My sample keys are:

CLIENT_HANDSHAKE_TRAFFIC_SECRET 24a0ba3ed6a030ec8e3dfaaf467fedf990caa19957f92d05a2a424a8a7ec0373 afd7382d4e2250b87110fea5381e69e557d4203aab16992d4b79c2b3a54bb6a4
SERVER_HANDSHAKE_TRAFFIC_SECRET 24a0ba3ed6a030ec8e3dfaaf467fedf990caa19957f92d05a2a424a8a7ec0373 7d4920ba247e0ce9eb43bd5f2bede6e015711131fc2a943e578a8ce260f38579
CLIENT_TRAFFIC_SECRET_0 24a0ba3ed6a030ec8e3dfaaf467fedf990caa19957f92d05a2a424a8a7ec0373 68b953f99d56afffb8bcdfbe8e14fa00a7da5a3df0fee6c6ed6efd20317acf9b
SERVER_TRAFFIC_SECRET_0 24a0ba3ed6a030ec8e3dfaaf467fedf990caa19957f92d05a2a424a8a7ec0373 9833238deb0bc979f8d2384da9c318297ae89b95cd88cfed1cf6dd27ba3f3873
EXPORTER_SECRET 24a0ba3ed6a030ec8e3dfaaf467fedf990caa19957f92d05a2a424a8a7ec0373 7b95217e262cb72a82241da2785d647ea4336215f9573b9a4fc657d05e91ee84

Decryption, Please!

Now, within Wireshark, go to Edit -> Preferences -> Protocols -> TLS and select your sslkey.log file at the “(Pre)-Master-Secret log filename”:

After hitting OK, Wireshark will decrypt all applicable TLS sessions directly. The following screenshot shows the original (encrypted) PCAP on the left-hand side, while the decrypted traffic on the right-hand side. While the original TLS traffic shows only “Application Data”, which is de facto random data, the decrypted part shows the actual protocol, an HTTP GET in this scenario:

On the Packet Bytes section (bottom right), you can switch between the original Packet vs. the Decrypted TLS view:

If you want to distribute the PCAP *with* the session keys, you can inject them into the capture file, which then turns into a *.pcapng file format: Edit -> Inject TLS Secrets:

Sample in the Ultimate PCAP

The sample of HTTPS traffic shown here, including the injected TLS keys, is part of the Ultimate PCAP. Please download and analyse it by yourself – the display filter to find those packets could be: http and tcp.port eq 443.

Follow {HTTP|TCP|TLS} Stream

Note the differences between the three “Follow …” methods. The well-known “Follow TCP Stream” will still show the encrypted data:

while the “Follow TLS Stream” shows the DEcrypted data, but still with the HTTP Content-Encoding such as gzip. Hence: still not completely readable (though decrypted):

Finally, the “Follow HTTP Stream“, when clicked on an HTTP packet (rather than on a mere TCP packet), brings us readable plain-text HTTP completely:

That’s it. Happy decrypting. ;) For further reading, have a look at this “SSL/TLS decryption: uncovering secrets” presentation by Peter Wu, at SharkFest’18 EUROPE, the Wireshark Developer and User Conference.

Soli Deo Gloria!

Photo by Markus Spiske on Unsplash.

After years of customers confusing Fortinet with Fortnite, the two companies finally decided to lean into the chaos. The result: FortiNite — a joint innovation designed to deliver “next‑gen latency acceleration” for Fortnite players worldwide, a groundbreaking collaboration with Epic Games’ Fortnite.

For years, support teams across the globe have endured phrases like “I can’t reach my Fortnite firewall” or “Is the Fortinet battle pass included?” Eventually, both companies realised:

What is FortiNite?

FortiNite is marketed as a “next‑generation, AI‑enhanced, cloud‑based latency reduction infrastructure accelerator”.

In plain English: FortiNite makes Fortnite faster. Much faster.

Leveraging Fortinet’s global Security Fabric and Epic’s galaxy of servers full of excited players, FortiNite promises:

🚀 Sub‑millisecond latency boosts
🔐 Ultra‑secure, accelerated gameplay tunnels
🎯 Skill‑based match acceleration mode
⚡ Adaptive Lag Prevention (ALP™)
🛡️ Anti‑Tilt Protection — because lag is the #1 cause of dramatic headset slams

Early internal benchmarks claim that FortiNite grants players “the equivalent of three additional shotgun reaction windows.”

Scientific? No.
Convenient? Absolutely.

Now with Full IPv6 Turbo Mode

One of FortiNite’s most celebrated features is its exclusive IPv6 Performance Pipeline.

Because while many ISPs still treat IPv6 like an optional DLC, FortiNite elevates it to its rightful place: the ultimate low‑latency gaming protocol (ULLGP).

FortiNite takes advantage of:

✨ Direct end‑to‑end IPv6 paths with fewer NAT dragons to slay
🚫 Zero NAT traversal — because NAT clearly stands for “Not Actually Tactical”
📈 Reduced congestion, thanks to IPv6 lanes that nobody else uses
💡 SmartFlow6™ — an AI engine that automatically reroutes packets away from congested IPv4 players (also known as “the lagging mortals”)

Fortinet humorously noted in their launch statement:

Epic Games added:

Integration with Existing Fortinet Products

True to the Forti‑ecosystem approach, FortiNite integrates seamlessly:

• FortiGate: Lights up in a “Victory Royale” pattern when latency dips below 5 ms
• FortiAnalyzer: Offers dashboards correlating your K/D ratio with packet‑loss spikes
• FortiManager: Lets admins distribute “No‑Build Mode” across entire VLANs
• FortiAP: Features a stealth SSID named SweatyLobby_5Ghz for peak tactical advantage

These features are, of course, “in beta” — a phrase which here means “completely fictional”.

Photo by ELLA DON on Unsplash.

You never stop learning. One topic that hadn’t crossed my path in the past decade is: Multicast. Whew. Alongside all the technical literature, online presentations, and various blog posts, I decided to approach it the classic way – through packet captures. ;)

So here’s a new part of the #UltimatePCAP, which contains quite a bit of PIM traffic, including Hello, Join/Prune, Register (via unicast!), and more. Of course, for IPv6 and legacy IP (IPv4). Let’s have a look:

Please note that I’m *not* a multicast expert. Luckily, there are many good resources out there. I recommend this Cisco Live presentation from Aleksandar Sofranic (YouTube): IP Multicast Introduction and Troubleshooting, or this (partially free) multicast course on NetworkLessons.com, or this blog post series from Emil Boklund.

Lab Setup

This is my lab, consisting of 2x Cisco router 2811 with IOS version 15.1(4)M12a and 1x Palo Alto Networks firewall PA-440 with PAN-OS 11.1.10-h1 (which only supports legacy IP for multicast, not IPv6). Everything is routed via OSPF/OSPFv3, while PIM is used for multicast traffic. The rendezvous point (RP) is statically configured on R1’s loopback address. A Raspberry Pi on the right-hand side is offering a multicast stream. Three clients are placed in the lab to receive the streams. The capture point was between the routers R1 and R2, leveraging a real TAP, namely the ProfiShark from Profitap.

Raspberry Pi ffmpeg stream (thanks to this Reddit posting):

ffmpeg -f lavfi -re -i "testsrc=size=640x360:rate=30:decimals=2" -c:v libx264 -f mpegts "udp://239.23.11.10:1234?pkt_size=1316"

Similar for IPv6:

ffmpeg -f lavfi -re -i "testsrc=size=640x360:rate=25:decimals=2" -c:v libx264 -f mpegts "udp://[ff05::2311]:1234?pkt_size=1316"

The clients/receivers used VLC to play the streams via udp://@239.23.11.10:1234, respectively udp://@[ff05::2311]:1234.

It’s Capturing Time!

This was the sequence of events that I captured for IPv4 on 2025-11-26. All times in UTC:

• 15:27 – cable between R1 and R2 was plugged into/through the TAP
• 15:31 – start of ffmpeg on the Raspberry Pi, sending to 239.23.11.10 on port 1234
  • 15:34 – client 1 starts listening to the multicast stream via VLC
  • 15:36 – client 1 stopped viewing (stream is cropped in the PCAP)
• 15:42 – stop of ffmpeg on the Raspberry Pi
  • 15:49 – client 1 starts again, though the stream is not present anymore
  • 15:50 – client 1 stops
• 15:51 – client 2 starts VLC, though the stream is not present
• 15:53 – client 2 stops

For IPv6, the following sequence was captured on 2025-12-03, all times are in UTC as well:

• 11:07 – cable between R1 and R2 was plugged into/through the TAP
• 11:10 – start of ffmpeg on Raspi, sending to [ff05::2311]:1234
  • 11:12 – client 3 starts listening to the multicast stream via VLC
  • 11:14 – client 3 stops
• 11:16 – stop of ffmpeg on the Raspi
  • 11:22 – client 3 starts again, though stream not running
  • 11:23 – client 3 stops

Some notes concerning the capture:

• I left only the beginning of the actual H.264 streams in there to keep the file as small as possible. (Can you decode them? ;))
• Neither IGMP (for IPv4) nor MLD (for IPv6) is interesting in this capture, as I captured between the two routers R1 and R2 rather than on the source or destination subnet of the multicast stream. Hence, we’re merely looking at PIM here.
• I don’t know why there’s PIMv1 traffic in there, since PIMv2 is the default at all.
• For IPv4, there are some unrelated PIM joins in there, as the clients requested some other multicast groups as well.
• Concerning IPv6, there’s a lot of ICMPv6 traffic in the capture, which relates to RS/RA, NS/NA, and MLD. I left them within the trace, as I left ARP for IPv4 there as well.

Wiresharking

Please download the UltimatePCAP by yourself in order to have a closer look at all those packets and sessions. The following screenshots give a rough overview, though.

5x for legacy IP, 5x for IPv6:

Shows & Configurations

Here are some multicast routing table outputs during the tests, all taken from R1:

As the stream started, but nobody was listening yet:

R1#show ip mroute

(*, 239.23.11.10), 00:00:25/stopped, RP 10.0.0.1, flags: SP
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list: Null

(192.168.124.12, 239.23.11.10), 00:00:25/00:02:34, flags: P
Incoming interface: FastEthernet0/1, RPF nbr 10.23.0.2
Outgoing interface list: Null

Client 1 listens to the stream:

R1#show ip mroute

(*, 239.23.11.10), 00:03:46/00:03:17, RP 10.0.0.1, flags: S
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    FastEthernet0/0, Forward/Sparse, 00:00:16/00:03:17

(192.168.124.12, 239.23.11.10), 00:03:46/00:01:13, flags: T
  Incoming interface: FastEthernet0/1, RPF nbr 10.23.0.2
  Outgoing interface list:
    FastEthernet0/0, Forward/Sparse, 00:00:16/00:03:17

Stream was not present anymore, client 2 requested it:

R1#show ip mroute

(*, 239.23.11.10), 00:00:38/00:03:27, RP 10.0.0.1, flags: S
  Incoming interface: Null, RPF nbr 0.0.0.0
  Outgoing interface list:
    FastEthernet0/0, Forward/Sparse, 00:00:38/00:03:27

Same for IPv6: Stream started, no one listening yet:

R1#show ipv6 mroute

(2A00:6020:AD0B:83C1:E37:DC1D:6FB8:B0DC, FF05::2311), 00:00:09/00:03:20, flags: SP
  Incoming interface: FastEthernet0/1
  RPF nbr: FE80::21A:6CFF:FEA1:2B98
  Outgoing interface list: Null

Client 3 listens to the stream:

R1#show ipv6 mroute

(*, FF05::2311), 00:00:37/never, RP 2A00:6020:AD0B:8399::1, flags: SCJ
  Incoming interface: Tunnel1
  RPF nbr: 2A00:6020:AD0B:8399::1
  Immediate Outgoing interface list:
    FastEthernet0/0, Forward, 00:00:37/never

(2A00:6020:AD0B:83C1:E37:DC1D:6FB8:B0DC, FF05::2311), 00:03:14/00:03:01, flags: SJT
  Incoming interface: FastEthernet0/1
  RPF nbr: FE80::21A:6CFF:FEA1:2B98
  Inherited Outgoing interface list:
    FastEthernet0/0, Forward, 00:00:37/never

Stream not present anymore, client 3 requesting it nevertheless:

R1#show ipv6 mroute

(*, FF05::2311), 00:00:02/never, RP 2A00:6020:AD0B:8399::1, flags: SCJ
  Incoming interface: Tunnel1
  RPF nbr: 2A00:6020:AD0B:8399::1
  Immediate Outgoing interface list:
    FastEthernet0/0, Forward, 00:00:02/never

For the sake of completeness, here are the configuration commands related to IP routing and multicast for both routers:

############
R1
############
ip cef
ip multicast-routing
ipv6 unicast-routing
ipv6 cef
ipv6 multicast-routing
!
interface Loopback0
 ip address 10.0.0.1 255.255.255.255
 ipv6 address 2A00:6020:AD0B:8399::1/128
!
interface FastEthernet0/0
 description Switch-gi1/0/39
 ip address 192.168.3.99 255.255.255.0
 ip pim sparse-mode
 ip ospf authentication message-digest
 ip ospf message-digest-key 1 md5 7 0331783E321E13567A053D170D175C3837
 ipv6 address 2A00:6020:AD0B:8303::99/64
 ipv6 ospf 6 area 0.0.0.0
 ipv6 ospf authentication ipsec spi 305419896 sha1 7 0327785F572B711B6B51415C35472F5A557A0D050D646D074B5F335B59750C0D0A715A563C47090A06
 bfd interval 999 min_rx 999 multiplier 3
!
interface FastEthernet0/1
 description Switch-gi1/0/40
 ip address 10.23.0.1 255.255.255.0
 ip pim sparse-mode
 ip ospf authentication message-digest
 ip ospf message-digest-key 1 md5 7 1422313E38151831102417273816502230
 ipv6 address 2A00:6020:AD0B:83C0::1/64
 ipv6 ospf 6 area 0.0.0.0
 bfd interval 999 min_rx 999 multiplier 3
!
router ospf 1
 router-id 10.0.0.1
 redistribute connected subnets
 network 10.23.0.0 0.0.0.255 area 0.0.0.0
 network 192.168.3.0 0.0.0.255 area 0.0.0.0
 bfd all-interfaces
!
ip pim rp-address 10.0.0.1
!
ipv6 pim rp-address 2A00:6020:AD0B:8399::1
ipv6 router ospf 6
 router-id 10.0.0.1
 bfd all-interfaces
 redistribute connected
!

############
R2
############
ip cef
ip multicast-routing
ipv6 unicast-routing
ipv6 cef
ipv6 multicast-routing
!
interface Loopback0
 ip address 10.0.0.2 255.255.255.255
 ipv6 address 2A00:6020:AD0B:8399::2/128
!
interface FastEthernet0/0
 description Switch-gi1/0/41
 ip address 10.23.0.2 255.255.255.0
 ip pim sparse-mode
 ip ospf authentication message-digest
 ip ospf message-digest-key 1 md5 7 133034273F1D36301F280C212F27443325
 ipv6 address 2A00:6020:AD0B:83C0::2/64
 ipv6 nd ra suppress all
 ipv6 ospf 6 area 0.0.0.0
 bfd interval 999 min_rx 999 multiplier 3
!
interface FastEthernet0/1
 description Switch-gi1/0/42
 no ip address
!
interface FastEthernet0/1.124
 description R2-Netz1
 encapsulation dot1Q 124
 ip address 192.168.124.1 255.255.255.0
 ip pim sparse-mode
 ipv6 address 2A00:6020:AD0B:83C1::1/64
!
interface FastEthernet0/1.125
 description R2-Netz2
 encapsulation dot1Q 125
 ip address 192.168.125.1 255.255.255.0
 ip pim sparse-mode
 ipv6 address 2A00:6020:AD0B:83C2::1/64
!
router ospf 1
 router-id 10.0.0.2
 redistribute connected subnets
 network 10.23.0.0 0.0.0.255 area 0.0.0.0
 bfd all-interfaces
!
ip pim rp-address 10.0.0.1
!
ipv6 pim rp-address 2A00:6020:AD0B:8399::1
ipv6 router ospf 6
 router-id 10.0.0.2
 bfd all-interfaces
 redistribute connected
!

Soli Deo Gloria!

Photo by 愚木混株 Yumu on Unsplash.

A rare use case on a Palo (at least from my point of view): Multicast Routing. And it can become as complex as you want. Fortunately, the basics are relatively easy to configure, at least if you have a rough understanding of multicast and routing with PIM and IGMP. (Recommended YouTube session here.) Let’s have a look at the common configuration steps on PAN-OS, the needed security policies to the special destination zone type of “multicast”, as well as some “show” outputs that can be used for troubleshooting:

The Lab

• My lab consists of 3x Cisco routers (2811, IOS Version 15.1(4)M12a) and 1x PA-440 “pa-lab” with PAN-OS 11.1.10-h1, ARE enabled, hence: logical routers.
• (Note that with PAN-OS 11.2 and ARE, multicast routing is not supported at all.)
• PIM-SM (sparse-mode) is used all over the lab. The rendezvous point (RP) is statically configured on R1’s loopback interface.
• Since Palo only supports legacy IP (IPv4) with multicast, IPv6 is not of interest in this setup. 😢
• A Raspberry Pi on the right-hand side offers a multicast stream at 239.23.11.10:1234. Accomplished by: ffmpeg -f lavfi -re -i "testsrc=size=640x360:rate=25:decimals=2" -c:v libx264 -f mpegts "udp://239.23.11.10:1234?pkt_size=1316".
• The Palo serves a client subnet directly (VLAN 51, client 1 aka receiver, IGMP), and is additionally connected to another “internal” router (R3, PIM), in which another receiver (client 2) resides.

The Config

The following screenshots provide an overview of the multicast settings needed for this kind of setup. Almost all settings were left at their defaults.

Special attention needs to be paid to the security policies:

• Allowing “pim” is only necessary if another router must communicate with the RP *through* the Palo, or if the Palo itself provides the RP. If the Palo is only terminating receivers (IGMP), there’s no PIM policy needed. (I’m not fully sure why, since for other routing protocols such as OSPF or BGP, those allow rules are a must. Yes, I’ve checked an explicit intrazone-deny rule as well. Still no hits.)
• IGMP must be allowed from the receiver’s zone to the special “multicast” destination zone. Of course, destination address objects can be used to further restrict the traffic.
• For the actual multicast traffic, in my case, a video stream on UDP port 1234, a policy has to allow this type of traffic from the multicast source (in my case: zone “transfer”) to the special “multicast” destination zone.

The Show

The following screenshots and CLI outputs were taken while both receivers were consuming the stream. (Using the VLC media player with udp://@239.23.11.10:1234.)

Session Browser during the stream: (the IGMP session was already gone)

Traffic Log *after* the receivers stopped consuming the stream:

Traffic Log for “pim”, in which router R3 contacted the RP:

Almost the same via the CLI show commands. One note, though: The show advanced-routing multicast pim state command is the only one that gives an output comparable to that from the Cisco world. E.g., only this one lists the Rendezvous Point for the (*,G) groups.

weberjoh@pa-lab> show advanced-routing multicast pim neighbor

Logical router: default

Interface                               Neighbor                 Uptime    Holdtime  Generation ID       DR Pri
ethernet1/1                             192.168.3.99             44:59:56  00:01:26  1170265741          1
ethernet1/5.131                         10.23.1.2                44:59:56  00:01:44  3086112133          1


weberjoh@pa-lab>
weberjoh@pa-lab>
weberjoh@pa-lab> show advanced-routing multicast igmp membership group 239.23.11.10

Logical router: default

Interface                               Address             Group               Source              Mode      Timer          Src  V    Uptime    Static
ethernet1/5.51                          192.168.51.1        239.23.11.10        *                   EXCLUDE   00:02:15       1    3    00:10:35  False
Total (Interface, Groups): 1


weberjoh@pa-lab>
weberjoh@pa-lab>
weberjoh@pa-lab> show advanced-routing multicast route group 239.23.11.10

Flags: S - Sparse, C - Connected, P - Pruned, M - SSM, R - SGRpt Pruned, F - FHR flag, T - SPT-bit set

Logical router: default

group          source              flags     Proto     incoming            outgoing            TTL  Uptime
239.23.11.10   *                   SC        IGMP      ethernet1/1         ethernet1/5.51      1    00:11:04
                                             PIM                           ethernet1/5.131     1    00:09:16
239.23.11.10   192.168.124.12      ST        IGMP      ethernet1/1         ethernet1/5.51      1    00:11:04
                                             PIM                           ethernet1/5.131     1    00:09:16
total route shown: 2


weberjoh@pa-lab>
weberjoh@pa-lab>
weberjoh@pa-lab> show advanced-routing multicast fib group 239.23.11.10

Logical Router:  default

maximum of mfib entries for this mfib:  275
number of mfib entries for this mfib:   3
number of mfib entries shown:           2

group            source           flags  incoming             outgoing
-----            ------           -----  --------             --------
239.23.11.10     0.0.0.0          1      ethernet1/1          ethernet1/5.131
                                                              ethernet1/5.51
239.23.11.10     192.168.124.12   2      ethernet1/1          ethernet1/5.131
                                                              ethernet1/5.51


weberjoh@pa-lab>
weberjoh@pa-lab>
weberjoh@pa-lab> show advanced-routing multicast pim state group 239.23.11.10

Logical router: default

(*, G):

group               RP                  up time   upstream join  join timer     RPF interface                           RPF next hop

239.23.11.10        10.0.0.1            00:11:41  Joined         00:00:20       ethernet1/1                             192.168.3.99/32

    oil interface                           local membership    join/prune          join expire timer   prune pending timer assert st           assert timer        assert winner addr  assert winner metric
    ethernet1/5.51      LOCAL               NOINFO              --:--               --:--               NOINFO              --:--               0.0.0.0             infinity

    ethernet1/5.131     NOINFO              JOIN                02:42               --:--               NOINFO              --:--               0.0.0.0             infinity

    pimreg              N/A                 JOIN                02:42               --:--               NOINFO              --:--               0.0.0.0             infinity

(S, G):

group               source              up time   upstream nbr        upstream join  join timer     RPF next hop        DR reg    DR reg stop timer   SPT

239.23.11.10        192.168.124.12      00:11:41  192.168.3.99        Joined         00:00:20       192.168.3.99/32     RegNoInfo --:--:--            False

    oil interface                           local membership    join/prune          join expire timer   prune pending timer assert st           assert timer        assert winner addr  assert winner metric
    ethernet1/5.131     NOINFO              JOIN                02:41               --:--               NOINFO              --:--               0.0.0.0             infinity


weberjoh@pa-lab>

Finally, this is how R3 looked like during the session:

R3#show ip pim neighbor
PIM Neighbor Table
Mode: B - Bidir Capable, DR - Designated Router, N - Default DR Priority,
      P - Proxy Capable, S - State Refresh Capable, G - GenID Capable
Neighbor          Interface                Uptime/Expires    Ver   DR
Address                                                            Prio/Mode
10.23.1.1         FastEthernet0/0          1d21h/00:01:39    v2    1 / G
R3#
R3#
R3#show ip mroute 239.23.11.10
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, B - Bidir Group, s - SSM Group, C - Connected,
       L - Local, P - Pruned, R - RP-bit set, F - Register flag,
       T - SPT-bit set, J - Join SPT, M - MSDP created entry, E - Extranet,
       X - Proxy Join Timer Running, A - Candidate for MSDP Advertisement,
       U - URD, I - Received Source Specific Host Report,
       Z - Multicast Tunnel, z - MDT-data group sender,
       Y - Joined MDT-data group, y - Sending to MDT-data group,
       V - RD & Vector, v - Vector
Outgoing interface flags: H - Hardware switched, A - Assert winner
 Timers: Uptime/Expires
 Interface state: Interface, Next-Hop or VCD, State/Mode

(*, 239.23.11.10), 01:19:52/stopped, RP 10.0.0.1, flags: SJC
  Incoming interface: FastEthernet0/0, RPF nbr 10.23.1.1
  Outgoing interface list:
    FastEthernet0/1.132, Forward/Sparse, 00:10:13/00:01:56

(192.168.124.12, 239.23.11.10), 01:19:51/00:02:34, flags: JT
  Incoming interface: FastEthernet0/0, RPF nbr 10.23.1.1
  Outgoing interface list:
    FastEthernet0/1.132, Forward/Sparse, 00:10:13/00:01:56

R3#
R3#
R3#show ip igmp groups 239.23.11.10
IGMP Connected Group Membership
Group Address    Interface                Uptime    Expires   Last Reporter   Group Accounted
239.23.11.10     FastEthernet0/1.132      00:10:33  00:02:42  192.168.132.11
R3#

The End

Uff. ;) That’s it for now. Of course, you can configure other scenarios, such as setting the Rendezvous Point on the Palo itself or using filters for various settings. But not for me this time.

Soli Deo Gloria!

Photo by Jeanson Wong on Unsplash.

The other day, I was troubleshooting some network-related stuff, using the built-in Packet Capture on a Palo Alto Networks firewall. And while it did the job at a first glance, I stumbled upon some packets that were simply not correct, read: were not present on the Ethernet cable at all and/or were missing some content.

Let’s have a short look. I was configuring OSPFv3 with authentication between a Palo NGFW and a Cisco router. I wanted to validate those authentication headers (IPsec AH) with the usage of a packet capture and Wireshark. Hence, I captured on the Palo itself, filtering on the ingress interface, at all four stages. Later on, I added a real TAP, my ProfiShark from Profitap:

The interesting part is: The receive stage (rx) showed OSPFv3 packets originated and sent by the Palo firewall itself (why in the rx stage?), missing the auth header that was present in the transmit stage! That is, there are two failures in this capture: Outgoing packets are captured in the rx stage (in other words: I would think that this packet was actually *received* by the firewall), and furthermore, those packets were NOT exactly those that were present on the tx stage at all. Here’s my side-by-side comparison of the RX/TX/TAP captures:

Q.E.D.

Is this intentional? Well, concerning the “receive stage”, the docs read: “When the packet is received on the dataplane processor.” Maybe those OSPFv3 packets are forwarded/transmitted/received internally from the control plane to the dataplane, so that they appear in the receive stage? Even with an “ingress interface” set as the capture filter.

(By the way: neither the drop nor the firewall stage showed any of those OSPFv3 packets. But that’s correct to my mind, since they aren’t dropped nor passing the firewall policies.)

Well-known TAP vendors are NEOX NETWORKS or Profitap, for example.

Don’t get me wrong. Packet captures on firewalls offer a quick and easy way to get an initial look at packets. In many cases, they are fully sufficient for troubleshooting layer 7 packets that simply pass through the firewall, where you just want to inspect the contents of a DNS query, for example.

However, as soon as you’re dealing with packets that are generated or modified by the firewall itself (routing protocols, NAT, IPsec, TLS interception, etc.), you can’t fully rely on these built-in packet captures. Unfortunately, this is exactly what happens regularly – to me as well.

For further reading, please consult at least parts 4 and 5 of Jasper‘s Network Capture Playbook: SPAN Port In-Depth and TAP Basics.

Soli Deo Gloria!

Photo by Ronda Dorsey on Unsplash.

I had a hard time figuring out how to configure OSPFv3 authentication on a Palo Alto Networks NGFW due to its different configuration formats compared to a Cisco router.

TL;DR: The SPI must be set in hexadecimal, while the actual key (40 chars, hexadecimal) must be grouped in 5 sections, separated with hyphens.

Talking about Cisco’s IOS, OSPFv3 authentication is set at the interface configuration level with the following command:

ipv6 ospf authentication ipsec spi <256-4294967295> sha1 <hey-string 40 chars>

e.g.:

ipv6 ospf authentication ipsec spi 305419896 sha1 CC41D07E889B5D610FAE78E88E88C559E45D1021

However, trying to set those values on Palo’s OSPFv3 Auth Profile, I encountered the following commit error: Error: Failed to parse IPSec manual-key tunnel/profile 'ah-sha1' authentication key.

Luckily, I wasn’t the first person struggling with this, hence DuckDuckGo led me to this post: “OSPFv3 Authentication Palo Alto to Cisco Router“.

The corresponding values to the above-mentioned example are:

SPI: 0x12345678
Key: CC41D07E-889B5D61-0FAE78E8-8E88C559-E45D1021

Configured under Network -> Routing -> Routing Profiles -> OSPFv3 -> OSPFv3 Auth Profile (using a PA-440 with PAN-OS 11.2.10, Advanced Routing Engine enabled):

(Of course, other hash algorithms than SHA-1 must be used, but my lab counterparts are not capable of it. ;))

Enabled either at the OSPFv3 – Area level:

OR at the individual Interface level (preferred from my point of view):

Now, the SPI is consistent with the representation in Wireshark. Note the “Authentication Header” within the IP header, as the OSPFv3 authentication leverages the IPv6 extension header for IPsec:

Speaking about the SPI, the hexadecimal format (Palo) makes more sense compared to the decimal format (Cisco). However, I find it rather nonsensical that you have to insert hyphens in the key. 🤦‍♂️

Final note: If you have set your intrazone-default policy (default: allow) to deny, you need explicit rules for OSPF to work. Little stumbling block here: with OSPFv3 authentication, you have to allow “ipsec-ah” in addition to “ospf” in order to work. Both the session table and the traffic log will show both applications (!), dependent on the originating node, which is either the Palo or the other routers on the network: (I’ve no idea where this “from port / to port” 20033 comes from.)

Soli Deo Gloria!

Photo by George Prentzas on Unsplash.

This post guides through a basic DNS tunneling setup with the usage of the appropriate tool “iodine“. It shows how DNS tunneling works and lists the commands needed to run this type of attack. That is, you can tunnel IPv4 packets through this DNS channel via the (internal) recursive DNS resolver! Nice approach. ;)

In the end, I’m pointing out how to block these tunnelling attempts with the DNS appliances from Infoblox, and the firewalls from Palo Alto Networks and Fortinet.

At first, let’s have a look at how DNS tunneling works in general:

For more information about iodine, the DNS tunneling tool of choice, have a look at their project homepage, or this more detailed blog post from David Hamann.

iodine Setup

For my tests, I used a delegated subdomain “io.weberlab.de”. That is: underneath my domain weberlab.de, I delegated (NS records) the subdomain “io” to the IP addresses of the server, which runs iodine:

io      IN      NS      lx3
lx3     IN      A       85.215.94.29
        IN      AAAA    2a01:238:4363:ee00:9169:a8a4:e572:d5f8

Server

• -f to run in foreground
• -c for checking disabled, to answer all incoming requests
• -P passphrase
• IP address of the internal tunnel interface <- that’s the fun part
• name of the delegated zone

weberjoh@h2877111:~$ sudo iodined -f -c -P passphrase 192.168.99.1 io.weberlab.de
Opened dns0
Setting IP of dns0 to 192.168.99.1
Setting MTU of dns0 to 1130
Opened IPv4 UDP socket
Listening to dns for domain io.weberlab.de

This creates a tunnel interface called “dns0”:

weberjoh@h2877111:~$ ip a s
[...]
4: dns0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1130 qdisc pfifo_fast state UNKNOWN group default qlen 500
    link/none
    inet 192.168.99.1/27 scope global dns0
       valid_lft forever preferred_lft forever

The “vendor” of iodine offers a checking tool at https://code.kryo.se/iodine/check-it/:

Client

• -f for foreground
• -P passphrase
• -r to NOT use the raw mode which would end in a direct connection to the server rather than the usage of the recursive DNS resolver
• IP address of the recursive DNS server
• name of the delegated zone used for this tunneling (I simply used an open DNS resolver found here)

weberjoh@vm32-test2:~$ sudo iodine -f -P passphrase -r 85.214.123.36 io2.weberlab.de
Opened dns0
Opened IPv4 UDP socket
Sending DNS queries for io2.weberlab.de to 85.214.123.36
Autodetecting DNS query type (use -T to override).
Using DNS type NULL queries
Version ok, both using protocol v 0x00000502. You are user #0
Setting IP of dns0 to 192.168.99.2
Setting MTU of dns0 to 1130
Server tunnel IP is 192.168.99.1
Skipping raw mode
Using EDNS0 extension
Switching upstream to codec Base64
Server switched upstream to codec Base64
No alternative downstream codec available, using default (Raw)
Switching to lazy mode for low-latency
Server switched to lazy mode
Autoprobing max downstream fragment size... (skip with -m fragsize)
768 ok.. 1152 ok.. ...1344 not ok.. ...1248 not ok.. ...1200 not ok.. 1176 ok.. 1188 ok.. will use 1188-2=1186
Setting downstream fragment size to max 1186...
Connection setup complete, transmitting data.

weberjoh@vm32-test2:~$ ping 192.168.99.1
PING 192.168.99.1 (192.168.99.1) 56(84) bytes of data.
64 bytes from 192.168.99.1: icmp_seq=1 ttl=64 time=25.9 ms
64 bytes from 192.168.99.1: icmp_seq=2 ttl=64 time=20.0 ms
64 bytes from 192.168.99.1: icmp_seq=3 ttl=64 time=22.8 ms
64 bytes from 192.168.99.1: icmp_seq=4 ttl=64 time=19.5 ms
^C
--- 192.168.99.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 19.513/22.054/25.927/2.552 ms
weberjoh@vm32-test2:~$ lynx ip.webernetz.net

To not only ping through this DNS tunnel, but to browse, I installed the proxy “squid” on the iodine server to surf through it. Now, I was able to use this proxy (behind the internal dns0 tunnel interface on the iodine server) to browse the Internet:

weberjoh@vm32-test2:~$ export http_proxy=http://192.168.99.1:3128
weberjoh@vm32-test2:~$ lynx ip.webernetz.net

This screenshot shows the CLI-based web browser “lynx”, which I used to open https://ip.webernetz.net to show my public IP. The insight: I’m online with the IPv6 address of the iodines server rather than the (private) IPv4 address of the client itself:

Q.E.D. 😂

This Wireshark screenshot shows a capture taken on the server component of iodine during the establishment phase. (It is also part of the Ultimate PCAP. Plz download and analyse it by yourself. Display filter: dns.qry.name contains "io2.weberlab.de".) You can see normal DNS queries and responses between the DNS resolver and the iodine server itself, while the queried names are mostly random:

Blocking DNS Tunneling Attempts

Always look on the blocking side of life. 🎶

Infoblox NIOS Recursive DNS

Using Infoblox NIOS as your recursive DNS server, tunneling events such as these are blocked after just a few packets. In my tests, the tunnel was detected and blocked just after four pings (right-hand side of the screenshot):

“DNS Tunneling detected: …” events are generated, and the relevant (sub-)domain is placed in the configured blocklist RPZ:

Palo Alto Networks NGFW

Using a next-gen firewall from Palo Alto Networks between the client and the Internet, the DNS tunneling attempts with “iodine” are blocked in various ways. In fact, it wasn’t easy at all to set up a DNS tunnel through the Palo Alto firewall. 😂 I had to disable several rules and profiles just to get it working for testing purposes. It was more due to the tool’s signature than to a generic detection of DNS tunneling.

1. I had an allow rule with the application “dns”, but iodine was detected as “tcp-over-dns”, hence: blocked.
2. Then I configured a port-based (service) allow rule for TCP/UDP port 53, but still with a security group with the anti-spyware strict profile: now connections were detected as “tcp-over-dns” (allowed), but recognised as a threat “Iodine DNS Tunnel Tool Command and Control Traffic Detection” -> blocked again. :) Nice.
3. Finally, the port-based (service) allow rule without any security profile made it.

In both blocking scenarios, the DNS tunnel was already blocked during the setup period!

weberjoh@nb15-lx:~$ sudo iodine -f -P passphrase -r 85.214.123.36 io.weberlab.de
Opened dns0
Opened IPv4 UDP socket
Sending DNS queries for io.weberlab.de to 85.214.123.36
Autodetecting DNS query type (use -T to override).
Using DNS type NULL queries
Retrying version check...
Retrying version check...
Retrying version check...
Retrying version check...
Retrying version check...
iodine: couldn't connect to server (maybe other -T options will work)

Fortinet Firewall

On a FortiGate firewall, a rule allowing DNS without a security profile allowed the attack (unlike Palo Alto, which is application-based by default and blocks a “non-DNS” attempt directly).

A rule that still allows DNS, but with the Application Control to block “Proxy”, blocks iodine as well. In this blocking scenario, the DNS tunnel was already blocked during the setup period! Very good. The application was detected as “Iodine”.

weberjoh@vm32-test2:~$ sudo iodine -f -P passphrase -r 85.214.123.36 io2.weberlab.de
Opened dns0
Opened IPv4 UDP socket
Sending DNS queries for io2.weberlab.de to 85.214.123.36
Autodetecting DNS query type (use -T to override).....................
iodine: No suitable DNS query type found. Are you connected to a network?
iodine: If you expect very long roundtrip delays, use -T explicitly.
iodine: (Also, connecting to an "ancient" version of iodined won't work.)

Soli Deo Gloria!

Photo by engin akyurt on Unsplash.

On the Internet, it’s not only “always DNS” – it’s also about securing DNS. DNS faces a wide range of attack vectors, each requiring different defensive strategies. Here comes an overview of DNS security, which gives you all the keywords at a glance.

This is my approach to a picture worth a thousand words. I use it during training sessions and at customers’ sites to explain concepts related to DNS security. Of course, this drawing isn’t perfect and doesn’t show every detail, but it’s a good starting point. ;) If you have any thoughts or corrections, please leave a comment below. (Thanks to Carsten Strotmann for reviewing it.)

Feel free to use it wherever you want. Please link back to this blog post.

If you want not only the poster but also a full session that explains all of this, take a look at my DNS security presentation (full video!) from SharkFest’25 EU.

Also, have a look at my at-a-glance version for mere DNS: It’s Always DNS – Poster and its corresponding presentation.

Soli Deo Gloria!

Photo by Mark James Panaligan on Unsplash.

I was presenting at the annual “Wireshark Developer and User Conference“, the SharkFest’25 EU, talking about “Securing DNS – Attacks and Defences“. It covered all the buzzwords related to DNS security, such as malware using DNS, DNS spoofing, DNS exfiltration & tunnelling, while defending them with the keywords as DNSSEC, DoH/DoT, feeds & blocklists, and so on.

Quite many techniques. ;) Luckily, the whole session was recorded. So if you’re interested, have a look!

Here are the slides (PDF):

At the end of the talk, I showed a summary of all DNS security attack vectors and countermeasures. If you’re interested in this “simple” overview, you can find it here.

If you have any comments or questions, please go ahead and use the comment function. ;)

Me in action 😂:

Photo by Peter Conrad on Unsplash.

While I was working on my presentation about “Secure DNS” for this year’s SharkFest, the Wireshark Developer and User Conference, I recognised that I’m still missing some DNS-related packet captures in the Ultimate PCAP, that is DNS over TLS and DNS over HTTPS. And while working on it with the DNSDiag toolkit (thanks, Babak!), I came across DNS over QUIC and DNS over HTTP/3. 😂 Here we go:

Please find the packets/sessions in the Ultimate PCAP with the following Display Filter, which filters for the used Do{T|H|Q|H3}-endpoints. For the sake of completeness, I also captured a standard UDP and TCP DNS session.

ipv6.addr in {2a13:1001::86:54:11:1,2a13:1001::86:54:11:201,2a10:50c0::ad1:ff,2a10:50c0::ad2:ff}

Also, refer to the packet comments of the first packet of each session, where I have listed the respective command.

Setup

For the DNS-servers, I primarily used the DNS4EU “Protective resolution” (UDP, TCP, DoT, DoH), while AdGuard DNS for DoQ and DoH3. Thanks to the DNSDiag tool “dnsping”, which supports all of those variants. (I used version 2.9.0 during my tests.) These were my six commands, each querying four times the A record of “heise.de”:

./dnsping.py --server 2a13:1001::86:54:11:1 -c 4 heise.de
./dnsping.py --server 2a13:1001::86:54:11:1 --tcp -c 4 heise.de
./dnsping.py --server protective.joindns4.eu --tls -c 4 heise.de
./dnsping.py --server protective.joindns4.eu --doh -c 4 heise.de
./dnsping.py --server dns.adguard-dns.com --quic -c 4 heise.de
./dnsping.py --server dns.adguard-dns.com --http3 -c 4 heise.de

Here’s the complete log:

weberjoh@nuc:~/dnsdiag$ ./dnsping.py --server 2a13:1001::86:54:11:1 -c 4 heise.de
dnsping.py DNS: [2a13:1001::86:54:11:1]:53, hostname: heise.de, proto: UDP, class: IN, type: A, flags: [RD]
42  bytes from [2a13:1001::86:54:11:1]: seq=1   time=8.661   ms  NOERROR
42  bytes from [2a13:1001::86:54:11:1]: seq=2   time=12.478  ms  NOERROR
42  bytes from [2a13:1001::86:54:11:1]: seq=3   time=11.645  ms  NOERROR
42  bytes from [2a13:1001::86:54:11:1]: seq=4   time=11.245  ms  NOERROR

--- [2a13:1001::86:54:11:1] dnsping statistics ---
4 requests transmitted, 4 responses received, 0% lost
min=8.661 ms, avg=11.007 ms, max=12.478 ms, stddev=1.646 ms
weberjoh@nuc:~/dnsdiag$
weberjoh@nuc:~/dnsdiag$
weberjoh@nuc:~/dnsdiag$ ./dnsping.py --server 2a13:1001::86:54:11:1 --tcp -c 4 heise.de
dnsping.py DNS: [2a13:1001::86:54:11:1]:53, hostname: heise.de, proto: TCP, class: IN, type: A, flags: [RD]
42  bytes from [2a13:1001::86:54:11:1]: seq=1   time=22.471  ms  NOERROR
42  bytes from [2a13:1001::86:54:11:1]: seq=2   time=24.437  ms  NOERROR
42  bytes from [2a13:1001::86:54:11:1]: seq=3   time=23.537  ms  NOERROR
42  bytes from [2a13:1001::86:54:11:1]: seq=4   time=22.864  ms  NOERROR

--- [2a13:1001::86:54:11:1] dnsping statistics ---
4 requests transmitted, 4 responses received, 0% lost
min=22.471 ms, avg=23.327 ms, max=24.437 ms, stddev=0.861 ms
weberjoh@nuc:~/dnsdiag$
weberjoh@nuc:~/dnsdiag$
weberjoh@nuc:~/dnsdiag$ ./dnsping.py --server protective.joindns4.eu --tls -c 4 heise.de
dnsping.py DNS: protective.joindns4.eu:853, hostname: heise.de, proto: TLS, class: IN, type: A, flags: [RD]
42  bytes from protective.joindns4.eu: seq=1   time=87.545  ms  NOERROR
42  bytes from protective.joindns4.eu: seq=2   time=91.599  ms  NOERROR
42  bytes from protective.joindns4.eu: seq=3   time=91.103  ms  NOERROR
42  bytes from protective.joindns4.eu: seq=4   time=90.454  ms  NOERROR

--- protective.joindns4.eu dnsping statistics ---
4 requests transmitted, 4 responses received, 0% lost
min=87.545 ms, avg=90.175 ms, max=91.599 ms, stddev=1.815 ms
weberjoh@nuc:~/dnsdiag$
weberjoh@nuc:~/dnsdiag$
weberjoh@nuc:~/dnsdiag$ ./dnsping.py --server protective.joindns4.eu --doh -c 4 heise.de
dnsping.py DNS: protective.joindns4.eu:443, hostname: heise.de, proto: HTTPS, class: IN, type: A, flags: [RD]
42  bytes from protective.joindns4.eu: seq=1   time=50.702  ms  NOERROR
42  bytes from protective.joindns4.eu: seq=2   time=50.802  ms  NOERROR
42  bytes from protective.joindns4.eu: seq=3   time=44.650  ms  NOERROR
42  bytes from protective.joindns4.eu: seq=4   time=45.645  ms  NOERROR

--- protective.joindns4.eu dnsping statistics ---
4 requests transmitted, 4 responses received, 0% lost
min=44.650 ms, avg=47.950 ms, max=50.802 ms, stddev=3.261 ms
weberjoh@nuc:~/dnsdiag$
weberjoh@nuc:~/dnsdiag$
weberjoh@nuc:~/dnsdiag$ ./dnsping.py --server dns.adguard-dns.com --quic -c 4 heise.de
dnsping.py DNS: dns.adguard-dns.com:853, hostname: heise.de, proto: QUIC, class: IN, type: A, flags: [RD]
42  bytes from dns.adguard-dns.com: seq=1   time=49.083  ms  NOERROR
42  bytes from dns.adguard-dns.com: seq=2   time=43.731  ms  NOERROR
42  bytes from dns.adguard-dns.com: seq=3   time=29.795  ms  NOERROR
42  bytes from dns.adguard-dns.com: seq=4   time=70.312  ms  NOERROR

--- dns.adguard-dns.com dnsping statistics ---
4 requests transmitted, 4 responses received, 0% lost
min=29.795 ms, avg=48.230 ms, max=70.312 ms, stddev=16.817 ms
weberjoh@nuc:~/dnsdiag$
weberjoh@nuc:~/dnsdiag$
weberjoh@nuc:~/dnsdiag$ ./dnsping.py --server dns.adguard-dns.com --http3 -c 4 heise.de
dnsping.py DNS: dns.adguard-dns.com:443, hostname: heise.de, proto: HTTP3, class: IN, type: A, flags: [RD]
42  bytes from dns.adguard-dns.com: seq=1   time=45.593  ms  NOERROR
42  bytes from dns.adguard-dns.com: seq=2   time=28.860  ms  NOERROR
42  bytes from dns.adguard-dns.com: seq=3   time=37.118  ms  NOERROR
42  bytes from dns.adguard-dns.com: seq=4   time=37.592  ms  NOERROR

--- dns.adguard-dns.com dnsping statistics ---
4 requests transmitted, 4 responses received, 0% lost
min=28.860 ms, avg=37.291 ms, max=45.593 ms, stddev=6.834 ms
weberjoh@nuc:~/dnsdiag$

Wiresharking all the Stuff

Since the payload is encrypted in all four cases, it’s only the headers (TCP/UDP destination ports) and the TLS handshake, if present, that are of interest. Depending on the variant, you can see the Server Name Indication (SNI) and the server certificate.

As with the current Wireshark version 4.6.0, DoQ is decoded as DTLS, while it should be QUIC. (Feature request is out. Feature request implemented after less than 24 hours.) As a workaround, use the “Decode As…” function, as used in the screenshot:

Traffic Logs on a Palo

Just because I was curious, those are the detected applications as seen from a Palo Alto Networks firewall (without TLS interception), PAN-OS 11.2.9, Application Version 9034-9733 (10/27/25):

It is not surprising that none of the DNS sessions were recognised as ‘DNS’, since the user data is completely encrypted. It is also interesting to note that direct DoQ traffic on the well-known UDP port 853 was only recognised as ‘unknown-udp’, while DoH3, which also runs via QUIC, albeit on the ‘classic’ QUIC port UDP 443, was recognised accordingly.

Thanks for watching. ;)

RFCs

• DNS over TLS: RFC 7858
• DNS over HTTPS: RFC 8484
• DNS over QUIC: RFC 9250
• DNS over HTTP/3: n/a

Appendix

Additionally, I added some more DoT and DoH packets to the Ultimate PCAP, originating from other tools: dig, as well as real user traffic from Firefox using DoH. Refer to the packet comments. For dig, I used the following commands:

dig @test2.weberlab.de netsec.blog +https
dig @test2.weberlab.de weberlab.de aaaa +dnssec +tls

Soli Deo Gloria!

Photo by Markus Winkler on Unsplash.

The other day, I was troubleshooting an issue where users reported that “some websites are working while some are not“. Uh. This is almost the worst scenario to face from a networker’s perspective. It’s way easier if things do or don’t work at all, but not this “some don’t” situation.

The scenario: Using Zscaler for outbound Internet connections, connected via a GRE tunnel from a Palo Alto Networks firewall. TL;DR: If it’s not DNS, it’s MTU. 😂 The “Suppress ICMP Frag Needed” option within the ICMP Drop section of the Zone Protection Profile did what it is meant to do: block “ICMP fragmentation needed” messages. Unfortunately, this killed *some* sessions which had the “Don’t fragment” bit set but exceeded the (lower) MTU of the GRE tunnel.

Troubleshooting such scenarios is quite awkward, since no log on the Palo GUI is helping. Neither the traffic log (connections are allowed) nor the threat log (no threat detected at all). You have to use either the well-known show counter global ... commands with some filters, or the show zone-protection ... information if you already suspect the zone protection to have its fingers in there, or some wild packet capturing and wiresharking. 😂 That’s what we did.

In the end, it was this “Suppress ICMP Frag Needed” within the Zone Protection Profile that was used on the *inbound* zone/interface from the client to the firewall. That is, this zone protection blocks *outgoing* ICMP destination unreachable – fragmentation needed messages that belong to previous *incoming* packets. Of course, we have disabled it again:

Here are some Wireshark screenshots from a now working connection. This is how the very first SYN packet appears on the wire: (Palo Packet Capture, transmit stage)

Here’s the whole session, showing the packet that was too big (1, later in the packet list due to an insufficient packet capture on the Palo itself, rather than with a real TAP), and two (!) ICMP destination unreachables – fragmentation needed messages (2, 3), lowering the “next-hop TTL” to 1400, respectively 1376 bytes:

Alternative Solution: Adjusting MSS

By the way: Another solution could have been adjusting the TCP MSS within the client’s interface on the Palo. The normal MSS = MTU – IP header – TCP header = 1500 – 20 – 20. But since a normal GRE header uses 24 bytes (another IP header with 20 bytes + GRE header of 4 bytes), the overall adjustment from the MTU should be 64 rather than the default of 40. (Same for IPv6, but now with an adjustment of 84 rather than 60.)

Disclore: I have not tested this approach yet. If anyone has, please write a comment!

The values would probably need to be adjusted further downward (i.e., a larger adjustment) if, for example, VLANs are used along the path or additional tunnelling mechanisms are in place. Test procedures would involve several lowering steps while capturing until no “ICMP fragmentation needed” errors are arriving anymore.

And I’m not even sure whether this approach would work or not, since the packet captures and screenshots above show that the client’s outgoing MSS was already quite low (1376), while the server still responded with exceeding payloads. 🤷🏻‍♂️

And this will only solve TCP problems, while the underlying problem of lower MTUs along the path is not solved.

Lessons Learned

• Path MTU Discovery (PMTUD) is still a thing!
• Blocking ICMP unreachables by default will break things. Allowing ICMP unreachables for previously allowed sessions is a best practice.
• Of course, gratuitous ICMP messages without formerly allowed connections that are sent from an attacker must still be blocked.
• The show zone-protection CLI command on a Palo helps troubleshoot IP connection failures that are beyond “denied by policy” or “detected threat”.

Further Reading/Watching

• ICMP ‘Destination Unreachable’ Messages
• Path MTU Discovery

Soli Deo Gloria!

Photo by Nick Wright on Unsplash.

I just ran into a partially working Palo Alto firewall — a PA-1410 shipped with PAN-OS 11.0.3-h10 and ZTP (Zero-Touch Provisioning) enabled — as I exited ZTP mode to configure the firewall in standalone mode. However, this config shortcut did not work as expected. :(

On the very first boot, I connected via the console port and exited ZTP mode:

Do you want to exit ZTP mode and configure your firewall in standard mode (yes/no)[no]?:yes
Warning: You have selected to provision the firewall in standard mode.
Do you want to continue (y/n)?:y
[  OK  ]

After that, I configured the management interface and performed a commit. However, some of the initial ZTP configuration snippets were still there — mostly within the service routes configuration, which I honestly didn’t expect at first glance. 🤦‍♂️

It took me quite some time to troubleshoot this issue, since I could reach the management interface via ping/HTTPS/SSH (hence layer 3 was working fine!), but had no outgoing connectivity at all — neither DNS nor ping. For example, a ping on the mgmt port ended up with this:

admin@PA-1410> ping host heise.de
ping: heise.de: System error

Asking the Internet brought me to this LIVEcommunity question (which talks about a PA-1410 as well 🤔) and to this KB article.

In the end, I simply ran this CLI command again:
set system ztp disable

The firewall immediately forced a reboot. After that reboot (and after changing the admin password again), the firewall was finally really in standalone mode.

Note that this KB article from PANW suggests the following settings in step 3, but I don’t know why:

set system setting template enable
set system setting template disable
set system setting shared-policy enable
set system setting shared-policy disable

In fact, the firewall already reboots after the set system ztp disable command. And both settings (template & shared-policy) are already enabled. Why should I disable them in the end, since I probably want to connect the firewall to a Panorama?

TL;DR: The initial PANW wizard doesn’t completely disable ZTP mode. You’ll need to run set system ztp disable manually — in any case!

Soli Deo Gloria!

Photo by Dustin Tramel on Unsplash.

For a few weeks, our PlayStation stopped downloading game updates. I figured it was just a temporary issue with the PS4. Since it didn’t affect me directly but only the kids, I didn’t pay much attention at first. I planned to wait for a firmware update from Sony. When such an update eventually came but didn’t solve the issue, I started getting suspicious – especially when I found almost no relevant results online for the official error code, which reads “(HTTP Status Code : 416) (CE-40862-0)”.

After conducting further detailed searches, I finally came across a post in the Palo Alto Networks LIVEcommunity. That definitely caught my attention. If there’s one thing that sets my home network apart from most “normal” households, it’s the fact that I have a Palo Alto firewall running – not your average consumer-grade router. 😂

A real screenshot of the error message. More details about HTTP 416 are here. Unfortunately, the CE-40862-0 error is not listed on any official Sony PlayStation webpages.

The LIVEcommunity article pointed me to a setting under Device → Setup → Content-ID → Content-ID Settings: “Allow HTTP partial response.”

Indeed, I had unchecked that box a few weeks earlier. Why? Because while working on best practices for our Palo Alto landscape, we discussed this option and concluded that blocking partial HTTP responses shouldn’t be a significant issue – modern browsers surely handle those things.

Well… not quite. ;) Turns out the PlayStation relies on exactly this feature to download updates via HTTP (yes, plain unencrypted HTTP – I’m not even using TLS interception here). I probably wouldn’t have figured it out on my own if I hadn’t found that community post. I enabled that checkmark, and the updates are working again. (Using a PA-440 with PAN-OS 11.2.9.) Thank you, Internet! Luckily, I was able to explain to my kids that enabling this option on the Palo Alto firewall was ultimately for their own security. 😂

As a network engineer, I naturally captured the faulty connection from the PS4 (before re-enabling the option) with help from the ProfiShark. Here are two Wireshark screenshots of such a failed connection: (Wireshark Version 4.6.0)

Follow HTTP Stream:

And here are more details from the Palo Alto firewall. Apparently, the following counter is responsible for this type of HTTP range request: ctd_http_range_response

You can view it via CLI like this:

weberjoh@pa-home> show counter global | match range
ctd_http_range_response                39376        0 info      ctd       system    Number of HTTP range responses detected by ctd

Monitoring this counter (in my case via Checkmk, Raw Edition 2.4.0p11, custom API integration) looks like this. You can clearly see the peaks when the PlayStation tries to pull updates. Same for the time period during which I had the option disabled on the Palo. :)

Final note: Yes, a better approach to omit this problem rather than allowing “HTTP partial responses” globally would be an Application Override policy as recommended by PANW. But I was too lazy since I’m talking about my home network here. ;)

Soli Deo Gloria!

Photo by Brett Jordan on Unsplash.

This goes out to anyone who uses more than one Site-to-Site VPN tunnel between two locations that are secured by firewalls from Palo Alto Networks. Using two (or even more) VPN tunnels, you need an automatic way to failover the traffic flow from one VPN to the other in case of failures. Here’s how to accomplish that requirement:

Pre-Notes

• You need 2x Site-to-Site VPNs between two firewalls. There may be 2x different ISPs on both sides, but at least on one side. Each ISP connection needs its own logical/virtual router to have its own default route.
• All tunnel interfaces on both sides must use IP addresses.
• All tunnel interfaces can remain in the same and single LR/VR. That is: Even though different LR/VRs are used for the ISP connections and the tunnel establishment, the final tunnel interfaces can be in the same LR/VR.
• You need 2x static routes to the same destination through those 2x different tunnel interfaces. The “primary” VPN should use a more preferred metric (i.e., lower value). This must be consistent on both sides! If not, you’ll run into asymmetric routing problems as both VPN tunnels will be up most of the time.
• Using a monitor profile with an action of “Fail Over” within the IPsec tunnel configuration takes a non-working VPN tunnel interface down. –> The (formerly preferred) static route will be kicked out of the forwarding table. ✅
• Of course, you could use a dynamic routing protocol such as OSPF or BGP to accomplish the same, while having a more complex overhead.

Some notes about my lab:

• My firewall “pa-home” on the left is connected to two different ISPs. Both offer dynamic IPv4 addresses only, hence I’m using a peer address type “dynamic” from the other firewall. Different local IDs are used to get those two tunnels authenticated. Terminates various L3 subnets; the summary route to this location simply sets 192.168.0.0/16 as the destination.
• (1x DS-Lite with CGNAT over fibre from “Deutsche Glasfaser” (DG), using DHCP and DHCPv6-PD, 1x Dual Stack over DSL from “Deutsche Telekom” (DTAG), using PPPoE and PPPoEv6. Though these details are not of interest here. :))
• Primary VPN through DTAG, tunnel.12, 172.16.12.0/30, routing metric of 20
• Secondary VPN through DG, tunnel.11, 172.16.11.0/30, routing metric of 50
• My firewall “pa-stg” on the right is behind a BGP-routed prefix (connected to 2x ISPs), with the advantage that it only has a single public IPv4 address, though high available due to BGP. Here, only one LR is used at all. Terminates a single L3 subnet: 192.168.4.0/24.
• pa-home is PA-440 with PAN-OS 11.2.9, Advanced Routing enabled
• pa-stg is a PA-460 with PAN-OS 11.2.4-h7, Advanced Routing enabled as well

The two routes on the pa-stg firewall are configured like this. No specials:

What if: Usage without any Monitoring

Just in case you’re interested: Without any monitoring profile set on those VPN tunnels, both routes will be present in the routing table (RIB), while the one with the better metric will be in the forwarding table (FIB). This is true even if one or both VPN tunnels are down!

Using a Monitor Profile with “Fail Over”

It’s fairly easy now: Just configure the tunnel monitor within each IPsec tunnel with a profile that has the action of “Fail Over” set, pinging the tunnel interface of the other side:

If one ISP connection or VPN tunnel fails, you’ll see the following “tunnel-status-down” system log, while the tunnel interface status in the IPsec tunnels section goes down (!), ending in only one route in the RIB as well as in the FIB:

This is how the user experience appeared. Also note the difference in the RTT, decreasing from 15 ms to 9-10 ms, depending on the ISP’s routing policies.

If the primary tunnel is up again, you’ll see a “tunnel-status-up” in the system log, while the primary route will be in the RIB/FIB again. Of course, on both sides of the VPN-tunnel simultaneously, since otherwise you would have asymmetric routing. Again, there are some pings lost, while the RTT (in my scenario) increased from 10 ms to 15 ms:

Final note: While I used the “Tunnel Monitor” section within the IPsec tunnels for this failover, you could also use the “Path Monitoring” within the static routes. It functions almost the same since it pings the other side of the VPN tunnel and only adds the routes in case of a working connection. However, to my mind, it’s more sound to get the tunnel interface down rather than simply deleting the route. I want this failover decision to be as close as possible to the IPsec tunnel configuration. But that’s an architectural choice.

Anyway. Happy failovering. ;)

Soli Deo Gloria!

Photo by Caleb Jones on Unsplash.

It’s really just a small thing, but very practical for me: In Wireshark, a feature request I submitted has been implemented. Now, when you click on an ICMP error, the corresponding (original) packet is highlighted.

Previously, clicking on a packet belonging to a flow would show all related packets, including any ICMP errors. However, if you selected an ICMP error packet itself, nothing happened. If you had many ICMP errors from different sessions, you had to go through the cumbersome process of figuring out which sessions they actually belonged to.

Now, you can simply scroll through the packet list as usual and immediately see whether related packets are present — and if so, which ones. Very handy.

The following screenshot shows two times the same PCAP, clicked on packet number 21, an ICMP TTL exceeded due to a previously issued traceroute. On the left-hand side with Wireshark version 4.4.8, nothing happens, while on the right-hand side with Wireshark version 4.5.0rc0, a line is shown which points to the packet that led to this ICMP error, namely packet number 5, the start of the traceroute.

Again, this proves that you can even contribute to Wireshark if you don’t have any coding skills, such as I.

Soli Deo Gloria!

Photo by Anika Huizinga on Unsplash.