As we have just set up a TLS capable syslog server, let’s configure a Palo Alto Networks firewall to send syslog messages via an encrypted channel. While it was quite straightforward to configure I ran into a couple of (unresolved) problems as I added and deleted some syslog servers and their certificates. Uhm.
Syslog & Certificate Configuration
I am using a PA-220 with PAN-OS 10.0.7 for this setup. My syslog-ng server with version 3.25.1 is running on a Ubuntu 20.04.3 LTS.
Basically, you just have to set the transport of a syslog server to “SSL”. (For whatever reason it’s still named SSL. It should read TLS since a couple of years.) I am using the “IETF” format nowadays. This seems to be the more modern standard, refer to RFC 5424 Appendix A.1. Though I don’t see any obvious differences between the BSD and the IETF type.
If you’re doing an immediate commit you will see these system logs:
|
1 |
Syslog SSL error while writing stream; tls_error='SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed' |
Screenshot:
–> This is because I am using a self-signed x.509 certificate on my syslog-ng server which is not trusted indeed. Good so far!
The same errors are appearing at the /var/log/syslog log file on the syslog-ng server itself:
|
1 2 3 4 |
Sep 30 08:30:19 vm32-test2 syslog-ng[6111]: Syslog connection accepted; fd='42', client='AF_INET6([::ffff:94.31.107.114]:43087)', local='AF_INET6([::]:6514)' Sep 30 08:30:19 vm32-test2 syslog-ng[6111]: SSL error while reading stream; tls_error='SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca', location='/etc/syslog-ng/conf.d/generic.conf:28:17' Sep 30 08:30:19 vm32-test2 syslog-ng[6111]: Error reading RFC6587 style framed data; fd='42', error='Connection reset by peer (104)' Sep 30 08:30:19 vm32-test2 syslog-ng[6111]: Syslog connection closed; fd='42', client='AF_INET6([::ffff:94.31.107.114]:43087)', local='AF_INET6([::]:6514)' |
You have to import the certificate of the syslog-ng server into the Palo as well:
Now it’s working. ;)
Note that for some reason the Palo does NOT use IPv6 for this outgoing syslog connection, though my FQDN had an AAAA record at the time of writing and the syslog server itself was accessible.
|
1 2 3 |
weberjoh@nb15-lx:~$ host test2.weberlab.de test2.weberlab.de has address 194.247.5.27 test2.weberlab.de has IPv6 address 2001:470:1f0b:16b0:20c:29ff:fea8:26f7 |
This is what the legacy IP TLS connection looks like with Wireshark:
Some maloperations?!?
Since I wanted to use IPv6, I configured the IPv6 address of the syslog-ng server manually:
The TCP handshake to the syslog server worked, but no TLS connection was established. I got the same errors in the system log:
And this is it in Wireshark:
After trying that I decided to go back to the FQDN of my syslog-ng server to let the Palo use it with legacy IP. However, I was not able to get it working again at all! I tried the following:
- without certificate -> not working
- imported certificate -> working
- deleted certificate -> still working (!!!???)
- change of the syslog server to a dummy
- change back to my real one -> not working (ok)
- imported certificate again -> still not working (not ok!)
- reboot of the whole device -> still not working
Hence I suppose some kind of bugs here. :(
Anyway, lessons learned:
Photo by Claudio Testa on Unsplash.
I have this same problem. We got 10.1 working with IPv4 and secure syslog to an external destination but were using the identity cert. Tried to swap to using the typical root CA or even intermediate so we would not have to upload the new identity cert every year but….. once we tried something new. It broke and did not ever connect even when we went back to known good. We have a ticket open with PA and I will provide an update here if I get a fix.
Reboot did not fix but now all the syslogs say successful even though it is failing. The cap shows our PAN resets after the ServerDone HS.
For Secure Syslog a few items not in PA docs…yet:
* Syslog Server Profile -> Syslog Server: MUST be FQDN to match cert. ( They should add SNI support with IPv4/IPv6 addresses in config IMHO)
* The “syslogng-ssl-conn-validation” must be set to skip at this time(no bug/feature ID known at this time)
—-
set syslogng-ssl-conn-validation explicit OCSP skip CRL skip EKU skip
set syslogng-ssl-conn-validation all-cons skip
—-
* Secure Syslog does NOT seem to work at Root or Intermediate CA validation. MUST have Identity Cert installed in store.
Confirmed this on PANOS 10.1.3-h1. Asked for PA to update Docs and add solution to community.
Johannes,
You could try setting to IPv6 address with a DNS entry and if needed disabling the validation checks for syslogng on PA.
We have the exact same problem as well. We got TLS syslog to work WITHOUT client authentication with the commands Johannes suggested:
set syslogng-ssl-conn-validation explicit OCSP skip CRL skip EKU skip
set syslogng-ssl-conn-validation all-cons skip
The message logs were complaining about our intermediate cert not having OCSP/CRL EKUs (which violate the RFC).
We can’t get Panorama to send a client certificate under any circumstances. We’ve tried many permutations on our rsyslog server. We have an existing Panorama 9.x installation that works perfectly, even with client cert auth.
We have a case open with Palo, but no joy. We’ve suggested that their TLS syslog implementation is broken.
The Syslog Server setting must match the case CN of the certificate even thought the RFCs say it should not matter. This is something I could not find in any documentation. I was able to replicate this on multiple versions over 9.x and 10.x. If you issue a certificate to CN=LogStash.mydomain.com and you put logstash.mydomain.com in the Syslog Server field the connection will fail. We still had to turn on all of the skip settings above. Change the Syslog Server field to LogStash.mydomain.com and it connects w/o issue.