At SharkFest’22 EU, the Annual Wireshark User and Developer Conference, I attended a beginners’ course called “Network Troubleshooting from Scratch”, taught by the great Jasper Bongertz. In the end, we had some high-level discussions concerning various things, one of them was the insight that TCP RSTs are not only sent from a server in case the port is closed, but are also commonly sent (aka spoofed) from firewalls in case a security policy denies the connection. Key question: Can you distinguish between those spoofed vs. real TCP RSTs? Initially, I thought: no, you can’t, cause the firewalls out there do a great job.
It turned out: you can!
Closed Port vs. Denying Firewall
I did a little lab for this. In a first step, I connected to one of my servers on port 21 – old-fashioned FTP. Since there was no FTP service on the server running, every connection attempt was immediately responded by a TCP RST – just as a normal Linux server behaves. The following screenshot shows five connection attempts (FileZilla) for IPv6 and five more for legacy IP, all with a RST answer from the real server:
In a second step, I configured my Palo Alto Networks firewall to block outgoing FTP connections and to respond with a RST to the client. This is common in internal environments, where you want your clients to have some sort of immediate feedback in case of blocked/closed connections. A silent drop from the firewall would result in a client application waiting for a timeout.
This screenshot shows the TCP RST that was perfectly spoofed and sent by the firewall:
If you swap between those two screenshots, you won’t see any obvious differences, don’t you? In fact, the source IP addresses (regardless of IPv6 or legacy IP) are spoofed by the firewall as if they were the real server. Good job.
How to distinguish between them???
We just proved that the packets look exactly the same, right? This is where Jasper gave a little hint at his session:
A picture is worth a thousand words:
Let’s have a look at some pings via IPv6 and legacy IP, especially on the hop limit / TTL of the returning packets which reveal the hop count of the returning route. For this, I added a custom column to Wireshark which displays the ipv6.hlim or ip.ttl:
That is: the returning IPv6 route is probably 10 hops away (since the remaining hop limit is 54, assuming the starting hop limit was 64), and the returning legacy IP route is probably 11 hops long.
Now let’s look at the RSTs again, focussing on the hop limit / TTL. This screenshot shows the 2x RSTs from the real server and 2x RSTs from the firewall:
Ha, I got you!
Q.E.D.
If you want to have a look at the captures by yourself, here we go: (They are not completely the Ultimate PCAP! Due to the bug from PAN-OS, see below, I decided to not include those falsified packets in the Ultimate PCAP. This relates only to the ICMPv6 unreachables source from the unspecified address “::”.)
Excursion: Palo Alto NGFW Policy Actions
I used a PA-220 with PAN-OS 10.2.1 for these tests. It was a little tricky to set the policies right for this. At first, blocking the application “ftp” did not work since the application detection of PAN will almost always let the first packets pass until it really detects the application. This is normal behaviour if you have other policies (further down) that are merely application based. My “FTP app deny” rule did not hit because the first SYN packet will flow through anyway, while the server is not responding (aka sending a RST), hence the PAN will log those connection attempts as “incomplete”:
That is: I used a port-based policy with TCP destination port 21 to block those connection attempts instantly. Now it was about selecting the correct type of “block” to have the firewall send TCP RSTs; refer to the documentation. The commonly used “Deny” did indeed block the connections, but did not respond with RSTs. (You will find those attempts in the pcap as well.) A “Drop” can be configured to “Send ICMP Unreachable” but no RSTs as well. (Note that the “Help” site on the WebUI itself states it differently, hence incorrect: “A TCP reset is not sent to the host or application unless you select Send ICMP Unreachable.”.) Finally, the “Reset client” did the trick. I also selected the “Send ICMP Unreachable” here, but I think it would have worked without it.
Here are the traffic logs during my tests:
- allowing FTP but the server sent a TCP RST
- action “deny”, that is: silently drop in this case
- action “drop” with sending ICMP unreach: silently dropping as well
- action “reset client” with sending ICMP unreach again. For whatever reason Palo does not highlight this policy action as red, which is a GUI bug in my point of view.
Appendix: What about UDP? –> ICMP
When it comes to UDP connections, there is no concept like a RST from the server within the same UDP session. (Remember: UDP is stateless.) Instead, the server is sending an ICMP port unreachable messages back to the client. Arrows 1 and 2 on the following screenshot. Note that the custom column in Wireshark displays two hop limits since the received ICMP packets (with their hop limits) embed the original packets that forced the error (with their own hop limits as seen by the server).
Talking about the Palo Alto Networks NGFW, it does *not* spoof the ICMP port unreachables, but sends ICMP administratively prohibited messages, clearly telling the client that there is a firewall in between. Those packets are sourced from the actual firewall IP addresses, arrows 3 and 4. Consistently, those ICMP packets have the hop limit / TTL from the egress interface, arrows 5 and 6:
BUT a BUG: Did you spot it? The Palo has an obvious bug, as it sends the ICMPv6 packets from the unspecified IPv6 of “::”. This definitely violates the standards. Nobody is perfect. ;) I opened a ticket and reported it, though I’m not expecting that they’ll fix it. [UPDATE] I was wrong, PANW has fixed it with PAN-OS 10.2.8, 11.0.3, and 11.1.0.[/UPDATE]
Soli Deo Gloria!
Photo by Kelly Sikkema on Unsplash.
I love your work! Packets never lie and you prove that :)
One thing I used for spotting firewall based resets (knowing the infra thing were happening in) is looking at RTT’s of the reset packet. If a server is at say 2 ms and you get a reset in 0.5 ms there might be something in between. Nevertheless this method is way more accurate and I will start using it for sure.
Nice post. I also figured this out a while ago when inspecting a pcap file involving firewall blocking malicious traffic.
Another technique I think also useful in some cases is to measure the latency of the received RST (which Wireshark can do by adding a delta column and/or set time reference). Let’s say you happen to know the server is 100 ms away, and you receive the RST within 10 ms. That should you tell something.
Thank you guys for your replies. On the other hand, I was not aware of your method.
Fun fact, as we’re talking about timing: You wrote the comments just a few minutes apart from each other, while each of you wrote basically the same not knowing, that someone else wrote the same at the same time. Nice.