Now that you have your own NTP servers up and running (such as some Raspberry Pis with external DCF77 or GPS times sources) you should monitor them appropriately, that is: at least their offset, jitter, and reach. From an operational/security perspective it is always good to have some historical graphs that show how any service behaves under normal circumstances to easily get an idea about a problem in case one occurs. With this post I am showing how to monitor your NTP servers for offset, jitter, reach, and traffic aka “NTP packets sent/received”.
During my work with a couple of NTP servers, I had many situations in which I just wanted to know whether an NTP server is up and running or not. For this purpose, I used two small Linux tools that fulfill almost the same: single CLI command while not actually updating any clock but only displaying the result. That is: ntpdate & sntp. Of course, the usage of IPv6 is mandatory as well as the possibility to test NTP authentication.
Yes, ScreenOS is end-of-everything (EoE), but for historical reasons I still have some of them in my lab. ;D They simply work, while having lots of features when it comes to IPv6 such as DHCPv6-PD. However, using IPv6-only NTP servers is beyond their possibilities. :(
Anyway, I tried using NTP authentication with legacy IP. Unfortunately, I had some issues with it. Not only that they don’t support SHA-1 but MD5, this MD5 key was also limited in its length to 16 characters. Strange, since ntp-keygen per default generates 20 ASCII characters per key. Let’s have a look:
I initially wanted to show how to use NTP authentication on a Pulse Connect Secure. Unfortunately, it does not support NTP over IPv6, which is mandatory for my lab. Ok, after I calmed down a bit, a configured it with legacy IP and got NTP authentication running. ;) Here’s how:
Configuring NTP authentication on the Infoblox Grid Master is quite simple. Everything is packed inside the single “NTP Grid Config” menu. You just have to enter the NTP keys respectively key IDs and enable authentication on the appropriate servers. In the case of incorrect authentication values an error message is logged. Very good, since this is not the case on some other network security devices (Palo, Forti).
Too bad that it only supports MD5 while SHA-1 should be used instead of.
A security device such as a firewall should rely on NTP authentication to overcome NTP spoofing attacks. Therefore I am using NTP authentication on the FortiGate as well. As always, this so-called next-generation firewall has a very limited GUI while you need to configure all details through the CLI. I hate it, but that’s the way Fortinet is doing it. Furthermore the “set authentication” command is hidden unless you’re downgrading to NTPv3 (?!?) and it only supports MD5 rather than SHA-1. Not that “next-generation”!
Finally, you have no chance of knowing whether NTP authentication is working or not. I intentionally misconfigured some of my NTP keys which didn’t change anything in the NTP synchronization process while it should not work at all. Fail!
Everyone uses NTP, that’s for sure. But are you using it with authentication on your own stratum 1 servers? You should since this is the only way to provide security against spoofed NTP packets, refer to Why should I run own NTP Servers?. As always, Palo Alto has implemented this security feature in a really easy way, since it requires just a few clicks on the GUI. (Which again is much better than other solutions, e.g., FortiGate, which requires cumbersome CLI commands.) However, monitoring the NTP servers, whether authentication was successful or not, isn’t implemented in a good way. Here we go:
This is how you can use NTP authentication on Cisco IOS in order to authenticate your external NTP servers respectively their NTP packets. Though it is not able to process SHA-1 but only MD5, you’re getting an authentic NTP connection. Let’s have a look:
Operating NTP in a secure manner requires the usage of NTP authentication, refer to my Why should I run own NTP Servers? blogpost. Using the Meinberg LANTIME NTP appliance with NTP authentication is quite simply since it requires just a few clicks. Even adding more and more keys (which requires manual work on any other Linux ntp installation) is done within clicks. That’s the way it should be.
As already pointed out in my NTP intro blogpost Why should I run own NTP Servers? it is crucial to leverage NTP authentication to have the highest trustworthiness of your time distribution all over your network. Hence the first step is to enable NTP authentication on your own stratum 1 NTP servers, in my case two Raspberry Pis with DCF77/GPS reference clocks.
When configuring a pool of NTP servers on a F5 BIG-IP load balancer you need to choose how to check if they are still up and running. There is no specific NTP monitor on a F5 BIG-IP that does an application layer health check (like there is for http or radius). The out-of-the-box options that can be used are only ICMP and UDP monitoring. Let’s first look at the pros and cons of using either (or both) of these monitors. Then let’s build a custom UDP monitor that does a better job at checking whether the NTP servers are still healthy.
As you hopefully already know, you should use at least three different NTP servers to get your time. However, there might be situations in which you can configure only one single NTP server, either via static IP addresses or via an FQDN. To overcome this single point of failure you can use an external load balancing server such as F5 LTM (in HA of course) to forward your NTP queries to one of many NTP servers. Here are some hints:
In case you’re responsible for an enterprise network or data center you should care about NTP. Refer to “Why should I run own NTP Servers?“. As a hobby technician you might first think about Raspberry Pis with self soldered GPS modules. Well, good idea to play with, but not reliable at all. Way to unstable, hard to update, no alerting, no service agreements, and so on.
Hence you should use a dedicated NTP appliance such as the Meinberg LANTIME NTP Time Servers. I am using a LANTIME M200 with a DCF77 correlation receiver in my lab. With this post I am showing how to set up this NTP server, giving some insights, and listing the advantages of such an appliance compared to a Raspberry Pi or any other DIY server approach. My wish list aka feature requests to this product round things up.
As always when you’re running own services you should update them regularly to have all known bugs fixed and security issues thwarted. Same for NTP servers based on Linux, as in my case running on Raspberry Pis. Especially when you’re actively joining the NTP pool project with your NTP servers you have to update them to the latest version of ntp since you might be misused for well-known DDoS attacks or other security related bugs.
So, what’s this all about? You can simply do an “apt-get upgrade”, don’t you? Well, unluckily the ntp packages within the Linux distributions are not always updated to the latest versions. Hence you need to compile the ntp software by yourself to have the latest release running. Still not that hard, though it requires a bit more attention.