The other day I wanted to verify whether a service running on my Linux server was listening on IPv6 as well as IPv4. It turned out that it wasn’t that easy to answer – if at all.
Which ports are in the listening state? –> For this question I used netstat -tulpen for a couple of years now. It turned out, that netstat is kind of deprecated and that I should use ss for this. Same options: ss -tulpen:
- t: TCP
- u: UDP
- l: listening
- p: show the corresponding process
- e: extended (important, see below)
- n: numeric, that is: not resolving IP addresses (DNS PTR records) nor service names (“22” instead of “ssh”)
So, what’s the deal now?
In my case, I was trying to verify whether or not syslog-ng is listening on both IPs. I already knew that my SSH daemon was listening on both protocols. So this was the output (sorry, you have to scroll horizontally):
|
1 2 3 4 5 |
weberjoh@nb17-lx2:~$ sudo ss -tulpen Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=1023,fd=3)) ino:22611 sk:3 <-> tcp LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=1023,fd=4)) ino:22615 sk:4 v6only:1 <-> tcp LISTEN 0 128 *:514 *:* users:(("syslog-ng",pid=1877,fd=15)) ino:1753628 sk:4f v6only:0 <-> |
–> At a first glance, it looks like ssh is running for IPv4 and IPv6, while syslog-ng shows only one line here. In fact, it IS listening on both Internet Protocols as well. Note the “v6only:0” at the end which implies that this socket is also capable of IPv4. The IPv6 line from ssh reads “v6only:1” at the end. (This is what the -e option is all about.)
In case you’re using netstat, this is the output there. It shows “tcp6” for syslog-ng which is even more inaccurate as it implies “only v6” to my mind:
|
1 2 3 4 5 6 |
weberjoh@nb17-lx2:~$ sudo netstat -tulpen Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 22611 1023/sshd tcp6 0 0 :::22 :::* LISTEN 0 22615 1023/sshd tcp6 0 0 :::514 :::* LISTEN 0 1753628 1877/syslog-ng |
It’s even getting weirder when listing IPv4 services only, because it does not show the “v6only:0” line at all, hence omitting the syslog-ng daemon completely, while it IS listening on IPv4:
|
1 2 3 |
weberjoh@nb17-lx2:~$ sudo ss -tulpen4 Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=1023,fd=3)) ino:22611 sk:3 <-> |
Twitter user Flüpke gave the following explanation: “Sockets bound to in6addr_any on Linux will receive IPv4 mapped traffic if the sockopt IPV6_V6ONLY is not set”. Uh, okay.
Note that this is not a specific “problem” from syslog-ng, but a generic approach on how to open sockets. Some years ago I encountered the same behaviour with ntopng.
By the way: Since I was really unsure whether the ports are open or not, I used Nmap from a machine in the same layer 3 subnet to check for both Internet Protocols:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
pi@pi05-random:~ $ sudo nmap -Pn -sS -p 514 194.247.5.6 Starting Nmap 7.40 ( https://nmap.org ) at 2021-05-20 14:40 CEST Nmap scan report for 194.247.5.6 Host is up (0.00066s latency). PORT STATE SERVICE 514/tcp open shell MAC Address: 00:21:70:B2:0E:6C (Dell) Nmap done: 1 IP address (1 host up) scanned in 3.76 seconds pi@pi05-random:~ $ pi@pi05-random:~ $ pi@pi05-random:~ $ sudo nmap -Pn -sS -p 514 -6 2001:470:1f0b:16b0::b17:22 Starting Nmap 7.40 ( https://nmap.org ) at 2021-05-20 14:41 CEST Nmap scan report for 2001:470:1f0b:16b0::b17:22 Host is up (0.00083s latency). PORT STATE SERVICE 514/tcp open shell MAC Address: 00:21:70:B2:0E:6C (Dell) Nmap done: 1 IP address (1 host up) scanned in 1.23 seconds |
To top it off
I wanted syslog-ng to listen on UDP and TCP on both Internet Protocols. Using this source statement:
|
1 2 3 4 5 6 7 8 9 10 |
source s_network { network ( ip-protocol(6) transport("udp") ); network ( ip-protocol(6) transport("tcp") ); }; |
I was able to get this:
|
1 2 3 |
weberjoh@nb17-lx2:~$ sudo ss -tulpen | grep syslog udp UNCONN 0 0 *:514 *:* users:(("syslog-ng",pid=20820,fd=15)) ino:2121626 sk:56 v6only:0 <-> tcp LISTEN 0 128 *:514 *:* users:(("syslog-ng",pid=20820,fd=83)) ino:2121627 sk:57 v6only:0 <-> |
But in the meantime, I recognized that my IPv4 machines (that send syslog message) were listed as “::ffff:192.168.3.53” and so on in my syslog folders. Twitter user Armin brought me to this RFC section “Compatibility with IPv4 Nodes”.
Then I tried to open the IPv4/IPv6 ports independently, which succeeded for UDP but not for TCP:
|
1 2 3 4 |
weberjoh@nb17-lx2:~$ sudo ss -tulpen | grep syslog udp UNCONN 0 0 0.0.0.0:514 0.0.0.0:* users:(("syslog-ng",pid=20887,fd=83)) ino:2123432 sk:58 <-> udp UNCONN 0 0 *:514 *:* users:(("syslog-ng",pid=20887,fd=15)) ino:2123431 sk:59 v6only:0 <-> tcp LISTEN 0 128 *:514 *:* users:(("syslog-ng",pid=20887,fd=84)) ino:2123433 sk:5a v6only:0 <-> |
Ouch. To my mind, this was incorrect anyway, since the second line shows “v6only:0” hence there were two different UDP sockets opened, both capable of IPv4.
Thanks to Twitter Users!
Thanks to my Twitter followers, who helped me out with this question. In the end, “I know that I know nothing”. Even after figuring it out for a couple of hours, I am not quite sure whether I understood it correctly… Any concerns from your site? Please leave a comment!
Photo by Franco Antonio Giovanella on Unsplash.
Johannes,
Thank you for documenting the inability of different applications to successfully document:
– Single application mapping to
– Multiple L2 mappings to
– Multiple L3 protocol(s) to
– Multiple L4 protocol and ports
The problems fall into three categories:
– The developer’s choice of library, sockets options code implementation and the trusting in existing code
– Different applications were developed based on older RFC’s and not rewritten to the IETF IPv6 standard 86. Also known as RFC8200.
– Confusion and inconsistency on how relationships map between the application, multiple interfaces, L2, L3, multiple L3 identifiers, and L4 protocols in a way that makes it easy to understand.
This impacts network and security applications which depend on these operating system commands. It also creates incomplete data representations, increasing the complexity and confusion in NOC’s, SOC’s and the help desk.
Little bit of background, for the last 18 years I have attended multiple network and security conferences, walking booth to booth discussing IPv6 findings on their products. I have now given up, providing free help to security companies including the unicorns.
Interesting enough, one of the companies attempted to contact me about how great their product is. A unicorn company which does not support IPv6 on their own Internet services, nor in their helpdesk documentation. Sad, spoke to them multiple times about the issue.
Again, thank you for being exposed this long ignored problem!
Joe Klein
I’d like to add that the default setting for ip6only is controlled via the kernel tunable net.ipv6.bindv6only (the default being 0).
The idea is to simplify configuration: many applications simply listening on port X will work dual-stack out of the box.
What catches many by surprise as well is that one has to maintain a dedicated list of firewall rules for IPv6, see ip6tables. If traffic is dropped by default in iptables, the same does not automatically apply to ip6tables.