Palo Alto PBF Problem

I migrated an old Juniper SSG ScreenOS firewall to a Palo Alto Networks firewall. While almost everything worked great with the Palo (of course with much more functionalities) I came across one case in which a connection did NOT work due to a bug on the Palo side. I investigated this bug with the support team from Palo Alto Networks and it turned out that it “works as designed”. Hm, I was not happy with this since I still don’t understand the design principle behind it.

However, it was a specific and not business critical case: One Palo Alto firewall with two ISP connections using a destination network address translation (DNAT, an old IPv4 problem) and policy based forwarding (PBF) with the same destination ports. Following are some more details:

(This problem appeared in April 2016. I was running a PA-200 with PAN-OS 7.1.0. After the support team told me that it works as designed, I built a workaround and did not follow this case anymore.)

The Problem

Refer to the following figure with the two different connections (red and green):

Red: Incoming connections from ISP1 to DMZ with DNAT. Green: Outgoing connections from DMZ to ISP2 via PBF.

The following NAT policy forwards incoming IPv4 connections (from ISP1 on eth1/1 for some destination ports, in my case 33004) to the private DMZ IPv4 address:

And the following PBF policy forwards outgoing IPv4 connections (from the DMZ) with the same range of destination ports (in my case again 33004) to ISP2 on eth1/2:

While outgoing connections through the PBF to eth1/2 worked anytime, incoming connections through the DNAT did NOT work while the PBF rule was enabled, see the “aged-out” connections:

The core problem was the returning path for incoming connections (red, eth1/1) which was hit by the PBF policy though it had NOT a destination port of 33004 but some other higher ports. This should not be the case. The rule was hit since the returning traffic had (of course) a source port of 33004. That is, the returning packets which should flow out of eth1/1 back to ISP1 were re-routed by the PBF rule to eth1/2. In fact, neither the NAT nor the PBF policy clearly specify which type of port (source or destination) it is using. There is only one single “service” that must be chosen, which should be always the destination port to my mind. (In case of the Juniper ScreenOS firewall there was a distinction between source and destination port.) [UPDATE] As Dmitry noted in the comment section (thanks!), the service objects are indeed configured with source and/or destination ports. However, my service objects were only configured with destination ports. That is, the hit by the PBF rule was still not correct. [/UPDATE]

The PAN support asked their engineering about this behaviour and gave me the following reply (which I don’t fully understand): “The reason we do not switch port when we do s2c direction pbf lookup is because we have some use cases that need pbf in s2c direction only. In these cases, we use either service type or destination port of c2s flow to do s2c lookup. That’s why the s2c flow hit the pbf rule.” This was captured under bug number 95540 but not fixed since it already works as designed.

Conclusion

Yes, almost everything that worked on a Juniper ScreenOS firewall (or any other firewall) also works on a Palo Alto firewall. But only “almost everything” since I had no problems with this same situation on the old ScreenOS firewall but only on the Palo.

Featured image: “Down Low” by André Hofmeister is licensed under CC BY-NC 2.0.

3 thoughts on “Palo Alto PBF Problem

  1. Johannes,

    Although I did not try to reproduce the problem you described and cannot add anything to the subject matter per se, I would like to point out that there is a clear notion of source and destination port of a service in PAN-OS. That is, you can define a service object with only a destination port specified, or with only a source port specified, or both. Then, when you use the service object in your NAT and / or PBF rules, the ports you define do matter.

  2. Johannes,

    I’ve run into you scenario before and also found the behavior odd as well. If I recall correctly I had two different workarounds. The first one would be a ‘no-pbf’ rule that matches the s2c return traffic. The other is an inbound PBF rule with symmetric return that matches your DNAT traffic. There are resource limitations per platform on symmetric return though, so I would lean toward option 1.

    Andy

Leave a Reply

Your email address will not be published. Required fields are marked *