NTP (Network Time Protocol) messages are sometimes rate-limited or blocked entirely by Internet operators. This little-known “NTP filtering” was put into place several years ago in response to DDoS (Distributed Denial of Service) attacks. NTP filtering may drop NTP messages based on rate or message size. Let’s dig into it:

I’ve monitored the performance of public NTP servers for a number of years. Time requests sent by NTP clients are unanswered for several common reasons.  The server may be down or overloaded. The time request or response may be lost due to network congestion. The NTP  server may throttle traffic, especially for clients making frequent requests. The server may be configured to accept only requests based on IP address ranges (e.g., geoblocking). Some ISPs, educational institutions or corporations, prohibit or limit NTP traffic. The client or server’s nearby Internet connection may store excessive in-flight IP packets, leading to bufferbloat. An NTP server’s administrator may have control of, or at least knowledge of, many of these factors. The operator can examine the server and the local network. They can ask their ISP whether a “no servers” policy is present. There’s more to this story, however.

Some History of NTP

The Internet can be an unfriendly place, especially for some traffic types.  NTP users should be aware of an intentional network impairment, often called “NTP filtering” that occurs today. This is deliberate dropping of NTP traffic by Internet operators. First some history. When NTP was first developed in the 1980s its primary author, David Mills, added many local/remote diagnostic capabilities to the early implementations. E.g., an NTP client could ask an upstream NTP server for detailed information about the server’s time sources. The diagnostics were divided into two groups: control messages (mode 6) and private messages (mode  7).  [The more common NTP client requests are denoted as mode 3, server responses are mode 4.] Some standardization efforts were made for NTP control messages, but not NTP private messages.

As Internet adoption increased so did many forms of abuse. One network abuse category is distributed denial of service (DDoS) attacks. The attackers send unwanted traffic from many sources (hence “distributed”) towards an IP destination potentially saturating access network bandwidth, or exhaust other computer memory and other resources. NTP’s diagnostic capabilities, particularly the monlist option, were often used beginning 2013/2014 as an attack vector. Monlist had two characteristics that appealed to attackers. A monlist request consists of single unauthenticated UDP message. Hence, source address spoofing was easily done, masking the actual DDoS source. Worse, the monlist response was typically much larger than the request making, this an amplification attack. This diagram from Cloudflare illustrates how the monlist attack works.

Monlist attack countermeasures were added to the major NTP software distributions so that servers running their updated distributions were not vulnerable to the spoofed monlist attack. This took time and is still incomplete. In 2020, six years after the large scale attacks, thousands of publicly accessible NTP servers are still vulnerable to the monlist attack.

NTP Filtering

The organizations responsible for Internet transport and security at ISPs/IXPs detected the escalating monlist attacks circa 2014 and devised solutions that placed restrictions on NTP traffic. This is often called NTP filtering. Two major solutions were developed depending on the organization and available equipment. One simple solution restricted NTP traffic based on message size. Normal NTP requests and responses have 48-byte lengths while the monlist responses were typically 440 bytes long. A sized-based NTP filter implemented by an ISP/IXP could block the monlist messages while permitting the normal requests and responses. These filters are still in place. In May and June 2020 I scanned several thousand servers with variable length NTP messages from an IPv4 client connected to an AT&T fiber service in suburban Chicago. The following graph shows the number of servers responding. A hard cutoff at NTP message size=428 bytes can be seen and is attributable to AT&T:

For a similar scan at the same location using an IPv4 client connected to a Comcast Internet service, see the following figure. Rather than a hard cutoff, there is a “hole” for NTP sizes between 212 and 460 bytes. The hole affects over one-third of the target servers.  Though less dramatic the hole can also be seen in the ISP=AT&T chart above:

The 212-460 byte hole seen in my tests is largely attributable to CenturyLink. Size-based blockage can also be seen within Europe, though it is less prevalent.

Size-based NTP blocking was discussed on the NANOG list and elsewhere several years ago. Years later these size-based blocks are still in place but do not affect normal NTP client-server traffic. However, the latest secure NTP time transfer protocol, Network Time Security (NTS), exchanges NTP packets with large authentication payloads. NTS interoperability tests have run afoul of the NTP size-based blocks. Workarounds are under discussion within the NTS community.

While size-based NTP filtering affects all traffic that meets certain size criteria, NTP rate limit filters cause some NTP traffic to be dropped. Here is an example. One of my Chicago-area NTP monitoring clients regularly queries the NIST NTP servers located in Gaithersburg, Maryland. The following figure shows that the success rate dropped from typically 100% to 20-40% in later October 2019. [Scheduled maintenance at Gaithersburg caused the low values on October 26, 27.]

What happened? The success rate seen from other NTP monitoring clients remained high.  Perhaps severe Internet congestion somewhere between Chicago and Gaithersburg existed. I explored that possibility. The NIST NTP servers also support an older time transfer protocol: TIME. Time uses UDP port 37 while NTP uses UDP port 123. Comparing the success rate for NTP and TIME was striking: NTP had poor performance as shown above which TIME was near-100% hence NTP traffic was being intentionally dropped.  Preferentially discarding NTP traffic is not a secret, see this presentation. Despite many inquiries, I’ve been unable to determine details such as exactly how NTP filtering is triggered, the profile of current NTP traffic that is considered malicious, or whether alternative means could be used to restrict NTP’s use in DDoS attacks. I was able to identify CenturyLink and several other ISP/IXPs as network operators implementing NTP rate-limiting.

Rate limiting causes problems for the NTP Pool. The NTP pool uses monitoring systems to check the pool’s NTP servers for accurate time and to verify that the servers have good network connectivity. An NTP rate limit filter can cause poor connectivity to some servers causing those servers to be temporarily excluded from the NTP pool.

Though I won’t go into details here, IPv6 NTP filtering is currently uncommon.

What happens next? After discussions with a number of Internet operators, I’ve seen little evidence that NTP filtering is necessary. Internet operators have little incentive to change the existing NTP filtering. For the most part, their customers are not complaining. A recent IETF draft proposes sidestepping NTP filtering by allocating an additional port for NTP in addition to the current UDP port 123.

Appendix: Discovering where NTP Filters are running.

The location of size-based NTP filters can sometimes be determined. Here I’m using mtr from one of my NTP test clients (ISP=AT&T). (Traceroute could also be used.) A 100 byte payload is sent on UDP port 123.

$ mtr -s 100 –udp -P 123 204.9.54.119
My traceroute [v0.93] Host
1. dsldevice
2. aa-bb-cc-dd.lightspeed.cicril.sbcglobal.net (AT&T)
3. 71.151.XX.YY (AT&T)
4. cr83.cgcil.ip.att.net
5. cgcil403igs.ip.att.net
6. 192.205.37.22
7. 50.248.118.166
8. ae4-2324.ls-transit.en180br.ord2.your.org
9. (waiting for reply)

Now we execute the same command with a 500 byte payload.

$ mtr -s 500 –udp -P 123 204.9.54.119
My traceroute [v0.93] Host
1. dsldevice
2. aa-bb-cc-dd.lightspeed.cicril.sbcglobal.net (AT&T)
3. (waiting for reply)

The NTP size filter is located between aa-bb-cc-dd.lightspeed.cicril.sbcglobal.net and 71.151.XX.YY, both AT&T routers.

Discovering where rate limiting is being done is more difficult. If the NTP filter is limiting NTP requests, mtr/traceroute may show where. Those tools won’t help if instead the NTP responses are being rate limited.

Photo by Daphné Be Frenchie on Unsplash.