This post shows a guideline for a basic installation of the open source syslog-ng daemon in order to store syslog messages from various devices in a separate file for each device.
I am using such an installation for my firewalls, routers, etc., to have an archive with all of its messages. Later on, I can grep through these logfiles and search for specific events. Of course it does not provide any built-in filter or correlation features – it is obviously not a SIEM. However, as a first step it’s better than nothing. ;)
Prerequisites
This tutorial relies on a blank Linux (server) installation such as shown here. I am using an Ubuntu server. I furthermore assume that the reader is aware of its devices that are capable of sending syslog messages. That is: I am only showing the syslog-ng installation and no further details on how to send syslog messages from various devices to the server.
Installation
The first step is to install the syslog-ng package. I am using an Ubuntu server:
|
1 |
sudo apt-get install syslog-ng |
[UPDATE] On a fresh 64-bit Ubuntu 14.04.2 LTS, I got an error while trying to install syslog-ng. The following answer found in the Internet works:
|
1 |
apt-get install syslog-ng syslog-ng-core |
[/UPDATE]
The default configuration file is /etc/syslog-ng/syslog-ng.conf. On Ubuntu, it has already a few lines that generate logfiles at /var/log/, e.g., the basic logfile “syslog”, which can be tailed with tail -f /var/log/syslog to see incoming messages from the system itself.
Configuration
I will now show the basic configuration of syslog-ng in order to:
- have an own folder for each device with
- a new file every day, nested in folders for year and month.
This requires a “source”, “filter” and “destination” which are then bound together. For more detailed configuration commands, this wiki from archlinux gives many good examples.
1) Source
Since the last line in the “syslog-ng.conf” config file ( /etc/syslog-ng/syslog-ng.conf) is @include "/etc/syslog-ng/conf.d/", all configuration files in the folder “conf.d” will be processed, too. Therefore, I generated a new configuration file called “firewalls.conf” in that subfolder sudo nano /etc/syslog-ng/conf.d/firewalls.conf. It has the following lines in it:
(Note: Replace USERNAME and USERGROUP with the name and group of the account from which the logfiles should be wrote to the disk such as “root” and “root”.)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
################################################## options { create_dirs(yes); owner(USERNAME); group(USERGROUP); perm(0644); dir_owner(USERNAME); dir_group(USERGROUP); dir_perm(0755); }; ################################################## source s_udp { network ( ip-protocol(6) transport("udp") port(514) ); network ( transport("udp") port(514) ); }; |
This “source s_udp” object is quite generic and simply listens on udp port 514 on both Internet Protocols (IPv6 and legacy IP) for incoming syslog messages. It must appear only once in the config file.
2a) Generic Destination
The simpliest way to generate a different folder for every device is to use the following destination (without a specific filter). This adds a folder with the “$HOST” attribute, which is most commonly the IP address of the logging device. With this few lines, no more filters/destinations are required to log from many devices. (Thanks to the comment from “T”, who ticked me to search for this solution):
|
1 2 3 4 5 6 7 8 |
################################################## destination d_host-specific { file("/var/log/firewalls/$HOST/$YEAR/$MONTH/$HOST-$YEAR-$MONTH-$DAY.log"); }; log { source(s_udp); destination(d_host-specific); }; |
or 2b) Specific Filter & Destination
Another way is to write a filter for incoming log messages in order to save them in a specific destination folder. Following is the template. The only lines to change are the two UPPER CASE variables without the $ sign (i.e., NAMEOFTHEFIREWALL and IPOFTHEFIREWALL):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
#Template for a new firewall in the firewalls.conf file #Entries to be changed: NAMEOFTHEFIREWALL and IPOFTHEFIREWALL ################################################## filter f_NAMEOFTHEFIREWALL { host("IPOFTHEFIREWALL"); }; destination d_NAMEOFTHEFIREWALL { file("/var/log/firewalls/NAMEOFTHEFIREWALL/$YEAR/$MONTH/$YEAR-$MONTH-$DAY.NAMEOFTHEFIREWALL.log"); }; log { source(s_udp); filter(f_NAMEOFTHEFIREWALL); destination(d_NAMEOFTHEFIREWALL); }; |
That is:
- the filter “f_NAMEOFTHEFIREWALL” filters upon the source IP address from the sending device,
- the destination “d_NAMEOFTHEFIREWALL” is set to the hierarchical path,
- and finally the “log” sequence takes any messages from the source and uses the filter to store into the destination path.
These few lines in the template can appear many times in the config file. (Remember: the source s_udp must appear only once.) So you can copy & paste it for every syslog device.
Final Restart
A restart of the syslog-ng daemon is required to have the just added configuration active:
|
1 |
sudo service syslog-ng restart |
After that, netstat -tulpen shows a few lines which reveal that the port 514 is listening on IPv6 and legacy IP:
|
1 2 3 |
weberjoh@jw-nb10-syslog-mirror:~$ sudo netstat -tulpen | grep syslog udp 0 0 0.0.0.0:514 0.0.0.0:* 0 6083099 5050/syslog-ng udp6 0 0 :::514 :::* 0 6083098 5050/syslog-ng |
That is, all devices are now logging into the syslog-ng server, in my case the /var/log/firewalls directory:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
weberjoh@jw-nb10-syslog-mirror:~$ ls -ahl /var/log/firewalls/ total 100K drwxr-xr-x 25 root root 4.0K May 9 10:50 . drwxrwxr-x 17 root syslog 4.0K May 9 06:39 .. drwxr-xr-x 4 root root 4.0K Jan 1 00:00 192.168.110.15 drwxr-xr-x 4 root root 4.0K Jan 1 00:00 192.168.110.20 drwxr-xr-x 4 root root 4.0K Jan 19 12:24 192.168.120.20 drwxr-xr-x 4 root root 4.0K Jan 4 10:05 192.168.120.7 drwxr-xr-x 4 root root 4.0K Jan 16 10:08 192.168.121.10 drwxr-xr-x 3 root root 4.0K Jan 13 12:18 192.168.121.20 drwxr-xr-x 4 root root 4.0K May 5 05:25 192.168.121.30 drwxr-xr-x 3 root root 4.0K May 9 10:42 2003:51:6012:120::2 drwxr-xr-x 3 root root 4.0K May 9 10:33 2003:51:6012:120::24 drwxr-xr-x 3 root root 4.0K May 9 10:35 2003:51:6012:120::25 drwxr-xr-x 3 root root 4.0K May 9 10:33 2003:51:6012:121::1 drwxr-xr-x 3 root root 4.0K May 9 10:48 2003:51:6012:121::10 drwxr-xr-x 3 root root 4.0K May 9 10:50 2003:51:6012:121::40 |
The paths are quite long but structured, e.g.:
|
1 |
/var/log/firewalls/fd-wv-fw02/2014/07/2014-07-21.fd-wv-fw02.log |
Examples
This is how syslog messages from a Palo Alto firewall look like when changing some policy rules:
|
1 2 3 4 5 6 |
Jul 21 16:14:01 192.168.120.2 1,2014/07/21 16:14:01,001234567891,CONFIG,0,0,2014/07/21 16:14:01,192.168.125.41,,edit,weberjoh,Web,Succeeded, vsys vsys1 rulebase security rules Ping Untrust Trust Deny,2429,0x0 Jul 21 16:14:58 192.168.120.2 1,2014/07/21 16:14:58,001234567891,CONFIG,0,0,2014/07/21 16:14:58,192.168.125.41,,edit,weberjoh,Web,Succeeded, vsys vsys1 rulebase security rules GlobalProtect an Untrust,2430,0x0 Jul 21 16:16:22 192.168.120.2 1,2014/07/21 16:16:22,001234567891,CONFIG,0,0,2014/07/21 16:16:22,192.168.125.41,,edit,weberjoh,Web,Succeeded, vsys vsys1 rulebase security rules SMTP ESA to Postfix,2431,0x0 Jul 21 16:18:06 192.168.120.2 1,2014/07/21 16:18:06,001234567891,CONFIG,0,0,2014/07/21 16:18:06,192.168.125.41,,edit,weberjoh,Web,Succeeded, vsys vsys1 log-settings profiles lfp-standard,2432,0x0 Jul 21 16:19:35 192.168.120.2 1,2014/07/21 16:19:35,001234567891,SYSTEM,general,0,2014/07/21 16:19:35,,general,,0,0,general,informational,"Commit job started, user=weberjoh, command=commit, client type=2, .",240518390879,0x0 Jul 21 16:19:35 192.168.120.2 1,2014/07/21 16:19:35,001234567891,CONFIG,0,0,2014/07/21 16:19:35,192.168.125.41,,commit,weberjoh,Web,Submitted,,2433,0x0 |
This are some Juniper ScreenOS logs during active Internet connections:
|
1 2 3 4 |
Jul 21 16:37:45 172.16.1.1 fd-wv-fw01: NetScreen device_id=fd-wv-fw01 [Root]system-notification-00257(traffic): start_time="2014-07-21 16:37:42" duration=3 policy_id=2 service=http proto=6 src zone=DMZ dst zone=Untrust action=Permit sent=304 rcvd=148 src=192.168.110.17 dst=62.138.108.130 src_port=53709 dst_port=80 src-xlated ip=192.168.110.17 port=53709 dst-xlated ip=62.138.108.130 port=80 session_id=6290 reason=Close - TCP FIN Jul 21 16:37:45 172.16.1.1 fd-wv-fw01: NetScreen device_id=fd-wv-fw01 [Root]system-notification-00257(traffic): start_time="2014-07-21 16:37:12" duration=33 policy_id=1 service=https proto=6 src zone=Trust dst zone=Untrust action=Permit sent=5699 rcvd=7283 src=192.168.125.41 dst=193.24.224.29 src_port=10221 dst_port=443 src-xlated ip=192.168.125.41 port=10221 dst-xlated ip=193.24.224.29 port=443 session_id=6023 reason=Close - TCP RST Jul 21 16:37:45 172.16.1.1 fd-wv-fw01: NetScreen device_id=fd-wv-fw01 [Root]system-notification-00257(traffic): start_time="2014-07-21 16:37:12" duration=33 policy_id=1 service=https proto=6 src zone=Trust dst zone=Untrust action=Permit sent=4947 rcvd=6531 src=192.168.125.41 dst=193.24.224.29 src_port=10219 dst_port=443 src-xlated ip=192.168.125.41 port=10219 dst-xlated ip=193.24.224.29 port=443 session_id=7902 reason=Close - TCP RST Jul 21 16:37:45 172.16.1.1 fd-wv-fw01: NetScreen device_id=fd-wv-fw01 [Root]system-notification-00257(traffic): start_time="2014-07-21 16:37:12" duration=33 policy_id=1 service=https proto=6 src zone=Trust dst zone=Untrust action=Permit sent=4707 rcvd=6297 src=192.168.125.41 dst=193.24.224.29 src_port=10220 dst_port=443 src-xlated ip=192.168.125.41 port=10220 dst-xlated ip=193.24.224.29 port=443 session_id=7667 reason=Close - TCP RST |
And this are some Cisco router log messages:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
May 9 08:30:49 2003:51:6012:121::2 : %SYS-5-CONFIG_I: Configured from console by weberjoh on vty0 (2003:51:6012:110::B15:22) May 9 08:30:50 2003:51:6012:121::2 : %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 2003:51:6012:120::10 port 514 started - CLI initiated May 9 08:31:01 2003:51:6012:121::2 : %IPV6_ACL-6-ACCESSLOGNP: list snmp-access-v6/10 permitted 41 2003:51:6012:120::13 -> ::, 10 packets May 9 08:32:25 2003:51:6012:121::2 : %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 192.168.120.10 port 514 stopped - CLI initiated May 9 08:32:44 2003:51:6012:121::2 : %SYS-5-CONFIG_I: Configured from console by weberjoh on vty0 (2003:51:6012:110::B15:22) May 9 08:36:01 2003:51:6012:121::2 : %IPV6_ACL-6-ACCESSLOGNP: list snmp-access-v6/10 permitted 41 2003:51:6012:120::13 -> ::, 10 packets May 9 08:41:01 2003:51:6012:121::2 : %IPV6_ACL-6-ACCESSLOGNP: list snmp-access-v6/10 permitted 41 2003:51:6012:120::13 -> ::, 10 packets May 9 08:46:01 2003:51:6012:121::2 : %IPV6_ACL-6-ACCESSLOGNP: list snmp-access-v6/10 permitted 41 2003:51:6012:120::13 -> ::, 10 packets May 9 08:46:51 2003:51:6012:121::2 : %IPV6_ACL-6-ACCESSLOGP: list vty-access-v6/10 permitted tcp 2003:51:6012:110::B15:22(56418) -> ::(22), 1 packet May 9 08:46:51 2003:51:6012:121::2 : %IPV6_ACL-6-ACCESSLOGP: list vty-access-v6/10 permitted tcp 2003:51:6012:110::B15:22(56418) -> 2003:51:6012:121::2(22), 1 packet May 9 09:16:27 2003:51:6012:121::2 : %IPV6_ACL-6-ACCESSLOGP: list vty-access-v6/10 permitted tcp 2003:51:6012:110::B15:22(56428) -> ::(22), 1 packet May 9 09:16:27 2003:51:6012:121::2 : %IPV6_ACL-6-ACCESSLOGP: list vty-access-v6/10 permitted tcp 2003:51:6012:110::B15:22(56428) -> 2003:51:6012:121::2(22), 1 packet May 9 09:16:33 2003:51:6012:121::2 : %RADIUS-4-RADIUS_DEAD: RADIUS server 2001:DB8::1812:1812,1813 is not responding. May 9 09:16:33 2003:51:6012:121::2 : %RADIUS-4-RADIUS_ALIVE: RADIUS server 2001:DB8::1812:1812,1813 is being marked alive. |
That’s it. ;)
Featured image: “Erika 9 typewriter” by dr. shordzi is licensed under CC BY-SA 2.0.
Thanks for the nice write up.
I’m in the process of implementing it…
Rod
Where do you make the filter/destination file? Is it a couple of lines you added to the first .log file or is it a completely different one?
An answer is much appreciated :)
It does not matter how many configuration files you have. I am using a single one “firewalls.conf” under the “/etc/syslog-ng/conf.d/” folder. This is the important point! You must store the file under this directory!
In this single “firewalls.conf” file I am using the filters and destinations etc. for a couple of devices.
The *.log files are created by syslog-ng. You MUST NOT edit them. These are the read-only log files.
Hope that helps?
Is this really the most efficient way of doing this? I have 100 Firewalls, apache servers and who knows what else that I am going to point to this thing and I rather not have to make 300 destination/filter/Log rules when I could just do it possibly by a few for each source type (Firewall, apache, etc)
Is there a better way to do that?
Yeah, great questions. Good idea. I just searched for that and modified the blog post and added the “Generic Destination” section. That should fit for you. ;)
Thanks, this is really useful, and work perfectly.
somehow l have to create the the folder /var/log/$HOST manually, else l will get the following error,
Jan 15 17:30:03 Test-virtual-machine syslog-ng[48022]: Error opening file for writing; filename=’/var/log/192.168.28.144/192.168.28.144-2016-01-14.log’, error=’No such file or directory (2)’
This is the folder privilege for the /var/log folder,
drwxrwxr-x 14 root syslog 4096 Jan 15 17:30 log
Can you share script to send email upon receiving certain event in syslog message received
Hi,
you must use a destination of type “smtp”, such as shown here:
https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/configuring-destinations-smtp.html
Hi Johannes,
I am using syslog-ng 3.0 to send logs to a SIEM I am using.
Much as I can see the OS logs in the SIEM, I do not see any application logs.
I want to discount syslog-ng as the fault, as I think the SIEM is failing to parse these logs.
Have you configured a DB server to send logs to a SIEM, if you could help me with this.
Thanks
IBM
Hi Isaac,
I am sorry, I am not using syslog-ng in exact that manner as you are doing it. I am only using it as a syslog forwarder, e.g., to a Dell CTA. That is, I have another destination with some options which is used for that:
destination d_DellCTA {
tcp(“10.10.10.10” port(1470) flags(“threaded”) template(“<$PRI>$R_DATE $SOURCEIP $MSGHDR$MSG\n”) template_escape(no));
Are you sure that your SIEM is the problem? Have you tested sending the syslogs directly to it? If your SIEM is the problem, what do you want to solve with syslog-ng?
thanks very much for sharing this , it helped me so much so implent syslog-ng in syslog server .
Hi Johannes,
I tried this on ubuntu14.04 and it works fine, but with ubuntu16.04 it isn’t is there any changes thats required to be made with 16.04, kindly help.
Hey captain,
I just verified my guide on a “Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-79-generic x86_64)” and it works without any errors. To install syslog-ng, simply type “sudo apt-get install syslog-ng” <- works fine in my lab. So I am sorry, but I don't know what exactly your problems are.
Good staff Johannes.
everything is running as expected and works fine so far
I’m using Ubuntu 14.04
I’m looking to have a nice GUI with filtering associated with syslog-ng which obviously makes it fabulous tool to search and create nice graphs.
have you came across this?
Hey NeMO.
I have no GUI options tested out so far. I am only using the CLI. Please have a look at this blogposts which summarizes some of the most useful bash commands in order to parse through logs: https://weberblog.net/2015/01/22/logfile-parsing/
Ciao.
I had to modify your directions as per:
https://unix.stackexchange.com/questions/261244/syslog-service-fails-to-start
In particular, I had to preface my edit command with ” LANG=C “:
LANG=C sudo vi /etc/syslog-ng/conf.d/FILENAME.HERE
It would appear that there is HTML code in your code snippets above. I presume it is the HTML color coding. When I added ” LANG=C “, there was no color in the file.