The other day, I was troubleshooting an issue where users reported that “some websites are working while some are not“. Uh. This is almost the worst scenario to face from a networker’s perspective. It’s way easier if things do or don’t work at all, but not this “some don’t” situation.

The scenario: Using Zscaler for outbound Internet connections, connected via a GRE tunnel from a Palo Alto Networks firewall. TL;DR: If it’s not DNS, it’s MTU. 😂 The “Suppress ICMP Frag Needed” option within the ICMP Drop section of the Zone Protection Profile did what it is meant to do: block “ICMP fragmentation needed” messages. Unfortunately, this killed *some* sessions which had the “Don’t fragment” bit set but exceeded the (lower) MTU of the GRE tunnel.

Troubleshooting such scenarios is quite awkward, since no log on the Palo GUI is helping. Neither the traffic log (connections are allowed) nor the threat log (no threat detected at all). You have to use either the well-known show counter global ... commands with some filters, or the show zone-protection ... information if you already suspect the zone protection to have its fingers in there, or some wild packet capturing and wiresharking. 😂 That’s what we did.

In the end, it was this “Suppress ICMP Frag Needed” within the Zone Protection Profile that was used on the *inbound* zone/interface from the client to the firewall. That is, this zone protection blocks *outgoing* ICMP destination unreachable – fragmentation needed messages that belong to previous *incoming* packets. Of course, we have disabled it again:

Here are some Wireshark screenshots from a now working connection. This is how the very first SYN packet appears on the wire: (Palo Packet Capture, transmit stage)

Here’s the whole session, showing the packet that was too big (1, later in the packet list due to an insufficient packet capture on the Palo itself, rather than with a real TAP), and two (!) ICMP destination unreachables – fragmentation needed messages (2, 3), lowering the “next-hop TTL” to 1400, respectively 1376 bytes:

Alternative Solution: Adjusting MSS

By the way: Another solution could have been adjusting the TCP MSS within the client’s interface on the Palo. The normal MSS = MTU – IP header – TCP header = 1500 – 20 – 20. But since a normal GRE header uses 24 bytes (another IP header with 20 bytes + GRE header of 4 bytes), the overall adjustment from the MTU should be 64 rather than the default of 40. (Same for IPv6, but now with an adjustment of 84 rather than 60.)

Disclore: I have not tested this approach yet. If anyone has, please write a comment!

The values would probably need to be adjusted further downward (i.e., a larger adjustment) if, for example, VLANs are used along the path or additional tunnelling mechanisms are in place. Test procedures would involve several lowering steps while capturing until no “ICMP fragmentation needed” errors are arriving anymore.

And I’m not even sure whether this approach would work or not, since the packet captures and screenshots above show that the client’s outgoing MSS was already quite low (1376), while the server still responded with exceeding payloads. 🤷🏻‍♂️

And this will only solve TCP problems, while the underlying problem of lower MTUs along the path is not solved.

Lessons Learned

• Path MTU Discovery (PMTUD) is still a thing!
• Blocking ICMP unreachables by default will break things. Allowing ICMP unreachables for previously allowed sessions is a best practice.
• Of course, gratuitous ICMP messages without formerly allowed connections that are sent from an attacker must still be blocked.
• The show zone-protection CLI command on a Palo helps troubleshoot IP connection failures that are beyond “denied by policy” or “detected threat”.

Further Reading/Watching

• ICMP ‘Destination Unreachable’ Messages
• Path MTU Discovery

Soli Deo Gloria!

Photo by Nick Wright on Unsplash.