The other day, I was troubleshooting some network-related stuff, using the built-in Packet Capture on a Palo Alto Networks firewall. And while it did the job at a first glance, I stumbled upon some packets that were simply not correct, read: were not present on the Ethernet cable at all and/or were missing some content.
Let’s have a short look. I was configuring OSPFv3 with authentication between a Palo NGFW and a Cisco router. I wanted to validate those authentication headers (IPsec AH) with the usage of a packet capture and Wireshark. Hence, I captured on the Palo itself, filtering on the ingress interface, at all four stages. Later on, I added a real TAP, my ProfiShark from Profitap:
The interesting part is: The receive stage (rx) showed OSPFv3 packets originated and sent by the Palo firewall itself (why in the rx stage?), missing the auth header that was present in the transmit stage! That is, there are two failures in this capture: Outgoing packets are captured in the rx stage (in other words: I would think that this packet was actually *received* by the firewall), and furthermore, those packets were NOT exactly those that were present on the tx stage at all. Here’s my side-by-side comparison of the RX/TX/TAP captures:
Q.E.D.
Is this intentional? Well, concerning the “receive stage”, the docs read: “When the packet is received on the dataplane processor.” Maybe those OSPFv3 packets are forwarded/transmitted/received internally from the control plane to the dataplane, so that they appear in the receive stage? Even with an “ingress interface” set as the capture filter.
(By the way: neither the drop nor the firewall stage showed any of those OSPFv3 packets. But that’s correct to my mind, since they aren’t dropped nor passing the firewall policies.)
Well-known TAP vendors are NEOX NETWORKS or Profitap, for example.
Don’t get me wrong. Packet captures on firewalls offer a quick and easy way to get an initial look at packets. In many cases, they are fully sufficient for troubleshooting layer 7 packets that simply pass through the firewall, where you just want to inspect the contents of a DNS query, for example.
However, as soon as you’re dealing with packets that are generated or modified by the firewall itself (routing protocols, NAT, IPsec, TLS interception, etc.), you can’t fully rely on these built-in packet captures. Unfortunately, this is exactly what happens regularly – to me as well.
For further reading, please consult at least parts 4 and 5 of Jasper‘s Network Capture Playbook: SPAN Port In-Depth and TAP Basics.
Soli Deo Gloria!
Photo by Ronda Dorsey on Unsplash.
Interesting observation, thanks for sharing! :) Your conclusion really hits the nail on the head. If you want to know the absolute truth of what’s happening on the wire, an external tap is often the tool of choice especially in complex scenarios.
By the way, your suspicion regarding the OSPF packets is spot on. Packets generated by the firewall’s control plane itself (as with routing protocols) are passed internally to the dataplane for processing. From the perspective of the DP, where the packet capture occurs, this internal hand-off looks like a “receive” event. This explains why they show up in the rx stage, even though they are outbound.
For anyone who wants to dive deeper into the details and recommendations, the official documentation offers good insights
https://docs.paloaltonetworks.com/ngfw/administration/monitoring/take-packet-captures
I’ve been reading your blog for years and am always impressed by the depth, quality and everyday relevance for firewall engineers of your posts. Keep up the great work! :)