An analysis of some falsified leap second warnings that appeared in November 2021 on public NTP servers out of the NTP Pool Project.
The NTP Pool is a volunteer organization that provides time synchronization service to hundreds of millions of computers worldwide. A typical client might query a particular NTP Pool server ~10-60 times/hour. Wikipedia lists some abusive clients that far exceeded the normal rate. This wastes NTP server resources, may interfere with other clients, and can trigger DDoS protections. In late 2019, a software update made some FortiGate firewalls very unfriendly to the NTP Pool.
NTP (Network Time Protocol) messages are sometimes rate-limited or blocked entirely by Internet operators. This little-known “NTP filtering” was put into place several years ago in response to DDoS (Distributed Denial of Service) attacks. NTP filtering may drop NTP messages based on rate or message size. Let’s dig into it: Continue reading NTP Filtering (Delay & Blockage) in the Internet
I did a short presentation at the spring 2020 roundtable of the UK IPv6 Council. The talk was about a case study I did with my NTP server listed in the NTP Pool project: For 66 days I captured all NTP requests for IPv6 and legacy IP while analyzing the returning ICMPv6/ICMPv4 error messages. (A much longer period than my initial capture for 24 hours.) Following are my presentation slides along with the results.
During my analysis of NTP and its traffic to my NTP servers listed in the NTP Pool Project I discovered many ICMP error messages coming back to my servers such as port unreachables, address unreachables, time exceeded or administratively prohibited. Strange. In summary, more than 3 % of IPv6-enabled NTP clients failed in getting answers from my servers. Let’s have a closer look:
Running your own NTP server(s) is usually a good idea. Even better if you know that they’re working correctly and serve their answers efficiently and without a significant delay, even under load. This is how you can use Wireshark to analyze the NTP delta time for NTP servers:
I am participating in the NTP Pool Project with at least one NTP server at a time. Of course, I am monitoring the count of NTP clients that are accessing my servers with some RRDtool graphs. ;) I was totally surprised that I got quite high peaks for a couple of minutes whenever one of the servers was in the DNS while the overall rate did grow really slowly. I am still not quite sure why this is the case.
For one month I also logged all source IP addresses to gain some more details about its usage. Let’s have a look at some stats:
Do you have a running NTP server with a static IP address? What about joining the NTP Pool project by adding your server to the pool? You will give something back to the Internet community and feel good about it. ;)
It doesn’t matter if you’re running a Raspberry Pi with GPS/DCF77 on your home, or a fully-featured NTP appliance such as the ones from Meinberg on your enterprise DMZ. Just a few clicks and your server will be used by the NTP Pool’s round-robin DNS. Here’s a simple tutorial:
Wherever you’re running an NTP server: It is really interesting to see how many clients are using it. Either at home, in your company or worldwide at the NTP Pool Project. The problem is that ntp itself does not give you this answer of how many clients it serves. There are the “monstats” and “mrulist” queries but they are not reliable at all since they are not made for this. Hence I had to take another path in order to count NTP clients for my stratum 1 NTP servers. Let’s dig in:
… since we all can use “pool.ntp.org”? Easy answer: Many modern (security) techniques rely on accurate time. Certificate validation, two-factor authentication, backup auto-deletion, logs generation, and many more. Meanwhile, we use an unauthenticated protocol (via stateless UDP) from unauthenticated sources (NTP pool) to rely on! Really?
If you are using a couple of different NTP sources it might be not that easy for an attacker to spoof your time – though not unfeasible at all. And think about small routers with VPN endpoints and DNSSEC resolving enabled, or IoT devices such as cameras or door openers – they don’t even have a real-time clock with a battery inside. They fully rely on NTP.
This is what this blogpost series is all about. Let’s dig into it. ;)