The NTP Pool is a volunteer organization that provides time synchronization service to hundreds of millions of computers worldwide. A typical client might query a particular NTP Pool server ~10-60 times/hour. Wikipedia lists some abusive clients that far exceeded the normal rate. This wastes NTP server resources, may interfere with other clients, and can trigger DDoS protections. In late 2019, a software update made some FortiGate firewalls very unfriendly to the NTP Pool.
Fortinet is a multinational corporation that produces computer security devices, including the FortiGate firewall. The firewall product documentation describes how to use the NTP Pool for time synchronization, e.g., by using:
1 2 3 |
0.europe.pool.ntp.org 1.europe.pool.ntp.org 2.europe.pool.ntp.org |
DNS will resolve 0.europe.pool.ntp.org into one of the hundreds of NTP servers in the NTP Pool. Let’s say that at firewall startup 1.2.3.4 was the NTP server selected by DNS. What happens if sometime later the administrator of 1.2.3.4 decides to stop participating in the NTP Pool and shuts down the public NTP service? We were told that older FortiGate NTP software did not handle that condition gracefully. A software update, FortiOS 6.2.3, was issued in December 2019 that periodically reissued the DNS lookup. In our example, if 0.europe.pool.ntp.org no longer resolved into 1.2.3.4 some re-initialization was done by the new software. That new code had a serious flaw. Bursts of NTP requests with rates that sometimes exceeded 20,000/sec were sent for up to 10 seconds duration. As DNS resolution changed, which is common for the NTP pools, a single FortiGate firewall would send bursts to different NTP pool servers.
A burst from a single FortiGate firewall is shown below
In early 2020, NTP pool operators noticed these distinctive bursts. They sometimes represented over 20 % of the total NTP traffic on a single server! Diagnostic work by several NTP Pool volunteers identified FortiGate firewalls as the source of these bursts. In April 2020, we contacted Fortinet support who acknowledged the bug and said that the next software release containing a fix would soon be deployed. However, the new software had unrelated bugs that prevented it from being widely deployed. On August 22, 2020, FortiOS 6.2.5 was released to eliminate the bursts.
The fix for FortiGate administrators is simple:
- Do not use FortiOS release 6.2.3, or
- if using release 6.2.3, do not use the NTP Pool.
FortiGate notified a handful of customers of the problem. However, as of this writing (November 2020), the NTP Pool is still receiving frequent NTP bursts from hundreds (at least) of FortiGate firewalls. While Fortinet was responsive in email and phone discussions, on the whole, their corrective actions were disappointing. Hopefully, FortiGate administrators will read this note and take corrective action as appropriate.
The diagnostic work was done by Miroslav Lichvar, Hal Murray, and Steven Sommars with contributions from several other NTP Pool volunteers.
Photo by Heather Zabriskie on Unsplash.
Hi Steven, any updates on this? Do you know if Fortigate finally fixed that in their latest firmware version?
Heiko,
According to Fortinet, the problem was fixed in early 2020, I cannot verify independently.
It is now September, 2021. We’re still seeing the NTP bursts worldwide. I do not understand why Fortinet isn’t activity working with their customers to correct the problem. There is a trivial fix.
Steve, thank you for your response. To me this looks like a pool abuse case. If you would describe this on LinkedIn, I would be happy to share it and tag Fortinet.
Heiko, I’m not on LinkedIn, but could contribute text.
As of 2021-09-19, the situation is getting worse in Lithuania. I’ve logged over 400 clients sending bursts to lt.pool.ntp.org. The majority (up to 90%) of NTP Pool requests from Lithuania are the bursts described here. Some of the abusive client IPs respond on TCP port 80 as Mikrotik running various versions of RouterOS. Most of these Lithuania clients come from ASN8764, ASN43627, ASN212046, ASN13194.
NTP bursts in other countries are still significant. New Zealand sees 15-20% of total NTP traffic coming from bursts. I estimate that 30% of NTP Pool requests in Austria comes from bursts.
I’ve send mail to many abuse@ mailboxes, but seldom see replies.
Steve, please contact me via email, happy to post an article again describing these pool abuses. I am sure Johannes can provide you with my email address or you follow and DM me on Twitter, if that is easier for you.