Palo Alto: User Group Count Exceeds Threshold

Palo Alto Networks, , , ,

We have run into an annoying situation: A hardware-dependent limit of user groups on a Palo Alto Next-Generation Firewall. That is: We cannot use more Active Directory groups at our firewalls. The weird thing about this: We don’t need that many synced groups on our Palo, but we have to do it that way since we are using nested groups for our users. That is: Palo Alto does not support nested groups out of the box, but needs all intermediary groups to retrieve the users which results in a big number of unnecessary groups.

I am asking you to give me some input on how you’re using user groups on the Palo. How are you using group filters? What count of AD groups do you have? Are you using nested groups (which is best practice)?

This is our main problem on a PA-5220: “User Group count of xyz exceeds threshold of 10000”:

Same on a PA-820, where the limit is 1000:

How we are using AD groups

TL;DR: Our users are member of container groups, which are member of different permission groups, e.g., “firewall” groups. We are using *only* such firewall groups in our Palo Alto policies, while users are only in the container groups.

A good article describing those nested groups permissions is here: Active Directory nesting groups strategy and implementation.

The Palo Problem

On a Palo, the user groups are synced from the Active Directory (LDAP profile) within Device -> User Identification -> Group Mapping Settings. The “Search Filter” limits the groups. In our case, we would *only* need our firewall groups. The “Group Member” attribute is set to “member” by default:

Using this setup, we are NOT retrieving users within our groups at all. :( In order to retrieve the users, we have to include the full path from our user groups to our firewall groups. (PAN: Nested User Groups in User-ID) In our case, this is:

Doing it that way, we now have all users listed within our firewall groups. Good so far. BUT:

Using about 2000 firewall groups in our PA-5220 (distributed among several VSYSes), we now need about 10000 groups to sync into the Palo for our users to belong to the right groups. This brings us to the hardware limit. :(

The key question is: Why is Palo Alto, read: the LDAP implementation, not able to crawl the users within nested groups without having to retrieve all intermediary groups?!?

(Not-working-) Ideas

We googled a lot, opened a support ticket and had several discussions with colleagues and Palo SEs. Along with packet captures and Wireshark deep dives to understand how the LDAP queries are working. ;) Here are a couple of ideas that are all not feasible for us:

  • Using a special LDAP search string for the group member attribute: “1.2.840.113556.1.4.1941“, LDAP_MATCHING_RULE_IN_CHAIN. We tried different settings such as “member:1.2.840.113556.1.4.1941:” or “(member:1.2.840.113556.1.4.1941:={0})”. Though we were able commit, no users came in.
  • Using the msds-memberTransitive attribute (link) for the group member. In theory a very good idea: Using this attribute for querying a group, the LDAP server returns all users that reside in subjacent groups. Nice! Indeed, this solved our issue. If Palo Alto didn’t have a bug right there. :( In fact, for every 11th group refresh, the Palo does a full sync which failed using this attribute. For some complex reasons. PAN engineering confirmed this erroneous behaviour, however, “this cannot be fixed”. Every 11th time the “msds-memberTransitive” is used for an LDAP search request but without specifying a base object. Hence the AD is not able to answer this extensive search and replies with an error: “00002120: SvcErr: DSID-03120451, problem 5012 (DIR_ERROR), data 592062”. Here’s how we used it:
  • Manually using the “Group Include List“. Sure, googling for our problem this is the first article which shows how to use the include list. In my opinion, this is ridiculous. We are using a couple of VSYSes with some more AD domains, having new (or obsolete) groups every now and then. Having to specify the relevant groups manually would be a major step back. Please note: We are using a Next-Generation firewall here!
  • Rebuilding our AD into another design without using nested groups. Well. Yeah. Thanks for the idea.

In the end, the support from PAN told us to ask for a feature request. Hm. Obviously, they are not accepting it as a bug, though engineering confirmed it…

Are we the only ones running into this?

Again, I am highly interested in how you are using groups in your AD and within your Palo policies. How do you remain below the threshold of user groups? –> Please write a comment below!

Appendix: Used CLI commands

Photo by Jonathan Ford on Unsplash.

4 thoughts on “Palo Alto: User Group Count Exceeds Threshold”

  1. Hi,
    bin auch schon vor einiger Zeit über das Problem gestolpert. Seitdem nutze ich einfach die Funktion “Group Include List” unter dem Punkt Device -> User Identification -> Group Mapping Settings. Anderer Weg, gleiches Ziel :-)
    Nachteil hierbei: Man muss die Gruppen erstmal durch die megalahme 10.0.X GUI da rein bekommen :-/

    Gruß
    Christian

    Reply

  2. With all the Palo Alto customers I have worked with, I have only seen the issue once.
    And it only occurred because the configuration was missing a search base which resulted in all groups being imported.

    To be honest, I have never seen a customer using that many “firewall relevant” groups (incl. nested groups).
    Even one customer that used 10+ VSYSs and lots of connected domains didn’t reach the limit of their PA (as far as I remember, it was a 5000 series).

    Is it really necessary in your scenario to use that many groups?

    Reply

  3. mmm – AD groups are like firewall objects and rules – you get asked to add them. You rarely get told when they are obsolete resulting in bloat. Trying to tidy up later and you run into risk averse Change Board – “is it working?” yes “then don’t touch it”.
    So over the years things grow – I’m now dealing with nearly 12,000 AD groups on our 5220 – it copes but not ideal. I’m in 12 of them – which after nesting comes to 180 :/
    So the other issue overtime is loopholes created by these which staff exploit without realising to do their legitimate business. These can be very difficult to close – it’s a big sprawling government organisation doing good deeds that no-one wants to trip up. At least now we’re trying to pull it back.

    Reply

  4. Hi, we had/have this issue (15000+ groups), but to get this to work you need to utilize Group Mappings and apply the Group Include List.

    This method works with Nested groups and the firewalls are still able to traverse the nested groups, pulling the user information of each nested groups. But the good part is the firewall does not included the nested groups in the group count thus not exceeding the limit.

    To make this work we ended up creating Palo Alto specific AD Groups (created a new OU to hold these groups). Then in in the Palo Alto groups we included the required nested groups.

    In the “Server Profile” tab we didn’t specify any Search Filters
    In the “Group Include List” tab, we added only the Palo Alto specific AD Groups we created.

    This meant we had to go through all our firewall rules, remove the existing user groups applied with the newly name Palo Alto groups. And we’ve been good ever since with 15000+ groups.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *