During my analysis of NTP and its traffic to my NTP servers listed in the NTP Pool Project I discovered many ICMP error messages coming back to my servers such as port unreachables, address unreachables, time exceeded or administratively prohibited. Strange. In summary, more than 3 % of IPv6-enabled NTP clients failed in getting answers from my servers. Let’s have a closer look:
I am participating in the NTP Pool Project with at least one NTP server at a time. Of course, I am monitoring the count of NTP clients that are accessing my servers with some RRDtool graphs. ;) I was totally surprised that I got quite high peaks for a couple of minutes whenever one of the servers was in the DNS while the overall rate did grow really slowly. I am still not quite sure why this is the case.
For one month I also logged all source IP addresses to gain some more details about its usage. Let’s have a look at some stats:
You have a running NTP server with a static IP address? What about joining the NTP Pool project by adding your server to the pool? You will give something back to the Internet community and feel good about it. ;)
It doesn’t matter if you’re running a Raspberry Pi with GPS/DCF77 on your home, or a fully-featured NTP appliance such as the ones from Meinberg on your enterprise DMZ. Just a few clicks and your server will be used by the NTP Pool’s round-robin DNS. Here’s a simple tutorial:
If you are operating a public available NTP server, for example when you’re going to join the NTP Pool Project, you probably want to test whether your server is working correctly. Either with a one-off measurement from hundreds of clients or continuously to keep track of its performance. You can use the RIPE Atlas measurement platform (Wikipedia) for both use cases. Here’s how:
Monitoring a Meinberg LANTIME appliance is much easier than monitoring DIY NTP servers. Why? Because you can use the provided enterprise MIB and load it into your SNMP-based monitoring system. Great. The MIB serves many OIDs such as the firmware version, reference clock state, offset, client requests, and even more specific ones such as “correlation” and “field strength” in case of my phase-modulated DCF77 receiver (which is called “PZF” by Meinberg). And since the LANTIME is built upon Linux, you can use the well-known system and interfaces MIBs as well for basic coverage. Let’s dig into it:
Beyond monitoring Linux OS and basic NTP statistics of your stratum 1 GPS NTP server, you can get some more values from the GPS receiver itself, namely the number of satellites (active & in view) as well as the GPS fix and dilution of precision aka DOP. This brings a few more graphs and details. Nice. Let’s go:
Now that you’re monitoring the Linux operating system as well as the NTP server basics, it’s interesting to have a look at some more details about the DCF77 receiver. Honestly, there is only one more variable that gives a few details, namely the Clock Status Word and its Event Field. At least you have one more graph in your monitoring system. ;)
Wherever you’re running an NTP server: It is really interesting to see how many clients are using it. Either at home, in your company or worldwide at the NTP Pool Project. The problem is that ntp itself does not give you this answer of how many clients it serves. There are the “monstats” and “mrulist” queries but they are not reliable at all since they are not made for this. Hence I had to take another path in order to count NTP clients for my stratum 1 NTP servers. Let’s dig in:
Now that you have your own NTP servers up and running (such as some Raspberry Pis with external DCF77 or GPS times sources) you should monitor them appropriately, that is: at least their offset, jitter, and reach. From an operational/security perspective it is always good to have some historical graphs that show how any service behaves under normal circumstances to easily get an idea about a problem in case one occurs. With this post I am showing how to monitor your NTP servers for offset, jitter, reach, and traffic aka “NTP packets sent/received”.
During my work with a couple of NTP servers, I had many situations in which I just wanted to know whether an NTP server is up and running or not. For this purpose, I used two small Linux tools that fulfill almost the same: single CLI command while not actually updating any clock but only displaying the result. That is: ntpdate & sntp. Of course, the usage of IPv6 is mandatory as well as the possibility to test NTP authentication.
It’s not always this simple DNS thing such as “single query – single answer, both via UDP”. Sometimes you have some more options or bigger messages that look and behave differently on the network. For example: IP fragmentation for larger DNS answers that do not fit into a single UDP datagram (hopefully not after the DNS flag day 2020 anymore), or DNS via TCP, or some newer options within the EDNS space such as “EDNS Client Subnet” (ECS) or DNS cookies.
I won’t explain any details about those options, but I am publishing a pcap with that kind of packets along with some Wireshark screenshots. Feel free to dig into it.
This is a list of missing features for the next-generation firewall from Palo Alto Networks from my point of view (though I have not that many compared to other vendors such as Fortinet). Let’s see whether some of them will find their way into PAN-OS in the next years…
Since my last blogposts covered many 6in4 IPv6 tunnel setups (1, 2, 3) I took a packet capture of some tunneled IPv6 sessions to get an idea how these packets look like on the wire. Feel free to download this small pcap and to have a look at it by yourself.
A couple of spontaneous challenges from the pcap round things up. ;)
Yes, I know I know, the Juniper ScreenOS devices are Out-of-Everything (OoE), but I am still using them for a couple of labs. They simply work as a router and VPN gateway as well as a port-based firewall. Perfect for labs.
For some reasons I had another lab without native IPv6 Internet. Hence I used the IPv6 Tunnel Broker one more time. Quite easy with the SSGs, since HE offers a sample config. But even through the GUI it’s just a few steps:
Of course, you should use dual-stack networks for almost everything on the Internet. Or even better: IPv6-only with DNS64/NAT64 and so on. ;) Unfortunately, still not every site has native IPv6 support. However, we can simply use the IPv6 Tunnel Broker from Hurricane Electric to overcome this time-based issue.
Well, wait… Not when using a Palo Alto Networks firewall which lacks 6in4 tunnel support. Sigh. Here’s my workaround: