… since we all can use “pool.ntp.org”? Easy answer: Many modern (security) techniques rely on accurate time. Certificate validation, two-factor authentication, backup auto-deletion, logs generation, and many more. Meanwhile, we use an unauthenticated protocol (via stateless UDP) from unauthenticated sources (NTP pool) to rely on! Really?
If you are using a couple of different NTP sources it might be not that easy for an attacker to spoof your time – though not unfeasible at all. And think about small routers with VPN endpoints and DNSSEC resolving enabled, or IoT devices such as cameras or door openers – they don’t even have a real-time clock with a battery inside. They fully rely on NTP.
This is what this blogpost series is all about. Let’s dig into it. ;)
Why am I starting a long series about NTP? Because I have many customers and colleagues who are way too lazy when it comes to NTP. “It’s working with some servers on the Internet, so why should I care?”
Why NTP matters
I basically want to cite this mail from the NANOG (North American Network Operators Group) mailing list:
“How about you set the time on your server ahead by 5 years. Got any idea what would happen?
- Most of your passwords would expire.
- All your SSL certs would expire.
- All your TOTPs, like Google Authenticator would fail.
- All your IPSEC tunnels would drop, and refuse to restart.
- Many of your cron jobs would got nuts, possibly deleting all your logs.
- Much of your DNSSEC would expire.
- Many of your backups would be deleted since they ‘expired’.
Until recently, setting your iPhone to 1 Jan 1970 would brick it. I’m sure there are many more examples, but likely you can no longer log in, via SSH or HTTPS, and your iPhone is dead. I think any of those would qualify as more than an annoyance.”
Think about it. ?
Want some more resources? Have a look at this paper: Bypassing HTTP Strict Transport Security [PDF] which almost only leverages a man-in-the-middle tool for NTP. Uh. Or this one: “The most common cause for secure connection errors turns out to be user systems having the wrong time“, Panos Astithas at Feeling safer online with Firefox.
And something from APNIC Labs – What’s the Time?.
: “What can we say about time and the Internet? The safest assumption is that most systems will be in sync with a UTC reference clock source as long as the definition of “in sync” is a window of 24 hours! If the ‘correct’ behaviour of an Internet application relies on a tighter level of time convergence with UTC time than this rather large window, then it’s likely that a set of clients will fall outside of the application’s view of what’s acceptable“, atGot a first hand experience on how critical NTP is today. A cluster farm had a couple machines get the wrong NTP data, whole thing went down because the clustering was so complex we just had to wait it out. 4 hours of down time due to a small NTP data packet corruption…
— Rob Fuller (@mubix) June 24, 2019
And what about the Internet of Things? IoT devices merely rely on NTP since they don’t have a built-in battery-powered real-time clock. That is: After booting they don’t have a time at all until they get one over NTP. Spoofing the NTP servers or packets would immediately mess up their time.
And one more example out of a networker’s daily business: Taking multipoint packet captures without synchronized times is a nightmare! You have to sync the time on your capture devices to have a chance to find your relevant packets all over your traces. Same for log messages: You need synced timestamps all over your network for accurate log correlation.
What to do? Usage of own Servers & Authentication!
You should use three on-site NTP servers as well as NTP authentication. That is:
- Use your own NTP servers inside your own network with different external clock sources. Advantages:
- You have your own stratum 1 NTP servers that are independent of other NTP instances on the Internet. Unknown NTP servers might be out of sync or not reachable over time.
- Using own antennas for external sources omits the dependency on single third parties. You should use at least three different external sources such as Galileo, DCF77, and GPS. That is: You should not rely on the time codes of a single foreign country.
- You can rely on your own hardware and its validity because it is not easy to alter them at all (as long as they reside in your own physically secured data centre). You can uniformly control your servers, such as inserting leap seconds.
- Your whole company gets almost exactly the same time since they’re using the same NTP servers throughout your enterprise which is not the case if every single device uses the NTP pool on its own which results in using hundreds of different NTP servers.
- All NTP connections are routed within your own network, hence no one on the Internet can spoof your packets. Furthermore, you have low network latency and jitter compared to routed packets through the Internet.
- Your own devices can alert you in case of failures, e.g. by mail, SNMP, syslog, info displays, and so on.
- Use three independent NTP servers. Why? Because of redundancy. One server might fail, hence you need at least two. However, if one out of two fails (that is: delivers a falsified timestamp) your clients don’t know which of those two sources to trust. Hence you should use at least three NTP servers. If one out of three fails, your clients are still able to get the correct time, since two correct sources win against one faulty. For further redundancy, you should position them in different physical locations as well as logical IP subnets.
- Use NTP authentication on your NTP servers and clients. With an authentication mechanism leveraging hash functions such as SHA-1 or using the newly developed NTS specification (network time security), your NTP clients can validate that they are talking to the correct source and are not spoofed by an attacker using a man-in-the-middle (MITM) attack. If you are using the older symmetric key version, you need to generate different keys (at least per security zone, better per device) on the server in order to configure the clients appropriately. If one key gets lost, you can replace it while not losing the security of all other keys/devices.
Blogpost Series w/ Raspberry Pis and Meinberg LANTIME
For this blogpost series, I primarily used a couple of Raspberry Pis. Simply because they are cheap, have small power consumption, and you can easily use the IO ports to connect other devices such as a GPS or DCF77 receiver modules. Works fine. However: You should NOT rely on those Pis in your production environment. Why? Because they don’t have a real-time clock which keeps their own time accurate. Hence they merely rely on their stratum 0 input. If this signal is lost, you will drift in time! Furthermore, they are not stable at all, hard to update (as you will see during my series) and everything needs to be configured by hand. Hence: Only use dedicated NTP appliances that are made to serve precise timestamps all over your network. For my posts, I am additionally using a Meinberg LANTIME M-200 device which I am comparing to the Pis at some points.
This blogpost series is split into three parts:
- Building stratum 1 NTP servers: How to use Raspberry Pis for NTP servers, advantages of purpose-built NTP appliances, load-balancing NTP, and so on.
- Using NTP authentication: Securing NTP packets with symmetric keys on several devices such as firewalls.
- Monitoring NTP servers: Getting telemetry data and stats from NTP servers.
All posts are related to my three IPv6-only stratum 1 NTP servers:
- ntp1.weberlab.de, a Raspberry Pi with a DCF-77 receiver,
- ntp2.weberlab.de, a Raspberry Pi with a GPS receiver and
- ntp3.weberlab.de, a Meinberg LANTIME M200 NTP Server with DCF77 and phase-modulated PRNG.
Feel free to use them via ntp.weberlab.de only. This FQDN will list only the IP addresses of those servers that are available. Of course you should only use them for test purposes, NOT for your enterprise, as this article is all about running your own servers. ;)
Throughout this series, I am only covering NTP but not PTP (Precision Time Protocol). Furthermore, I am not attacking NTP via MITM attacks (at least not now) but only show how to prevent them with NTP authentication, nor do I attack the external time sources DCF77 or GPS. And I am not diving into DDoS attacks that use NTP servers (reflection & amplification attack) since I assume that you’ll primarily use your servers only within your own network and not on the public Internet.
Further Reading
As always you can read even more. ;) If you’re interested in the deep details about the network time protocol itself you should have a look at the Internet Protocol Journal volume 15, number 4 [PDF] and its article “Protocol Basics: The Network Time Protocol” from Geoff Huston. You should also have a look at this Recent NTP papers website with some really deep-dive papers about NTP. Wow. For example this one: Challenges in Time Transfer using the Network Time Protocol (NTP).
Anyway, here are some more links:
- RFC 5905: Network Time Protocol Version 4: Protocol and Algorithms Specification
- RFC 7822: Network Time Protocol Version 4 (NTPv4) Extension Fields
- RFC 8633: Network Time Protocol Best Current Practices
- Time Synchronization Blog – NTP vs PTP: Network Timing Smackdown!
- Time Synchronization Blog – PTP: Why is IEEE 1588 so accurate?
- Meinberg – Network Time Protocol
- Ivan Pepelnjak – Network Time Protocol (NTP) In a Nutshell
- (de) PTB – Zeitsynchronisation von Rechnern mit Hilfe des “Network Time Protocol”
- RIPE Labs – NTP Measurements with RIPE Atlas
- Cloudflare System Status – Some DNS lookups causing 5xx errors due to leap second bug
- APNIC Labs – What’s the Time?
- Paper: Masters of Time: An Overview of the NTP Ecosystem [PDF]
- Wikipedia – NTP server misuse and abuse
- Building a more accurate time service at Facebook scale
As well as security-related stuff:
- Boston University – The Security of the Network Time Protocol
- ACM Queue – Securing the Network Time Protocol
- Hanno’s blog – Don’t update NTP – stop using it
Disclaimer
I am not set against the NTP pool project in general. It is a great community-driven project that I am using client-side (for many small devices such as routers, Raspberry Pis, IoT, …) as well as server-side since I am contributing with at least one online NTP server. However, my focus on this blogpost series is the enterprise network. And you should not use the NTP pool within your enterprise as at all, but own independent stratum 1 NTP appliances along with NTP authentication.
Featured image “Breitling Super Avenger” by W.A.Smith is licensed under CC BY-NC-ND 2.0.
Love it, but additional benefit you didn’t mention: combine your own NTP with a gateway NTP hijack rule so internal (compromised) devices could never participate in an outbound NTP reflection attack.
Both device types benefit:
1) Clean devices benefit from getting a *local* answer even if they try to get NTP update from time.apple.com, for example
2) Infected devices benefit the network owners in being unable to launch NTP reflection attacks.
Johannes, I would like to add that the same principles apply to any other UDP-based unauthenticated protocol, including DNS. Offer it locally, hijack/redirect port 53 and everybody is happier and safer.
Hallo Johannes,
in diesem Artikel erklärst du aus nachvollziehbaren Gründen, warum man 3 NTP-Server betreiben sollte.
Nehmen wir an ein Angreifer kann das DCF-77 Signal verfälschen (da gabs sogar mal ein Raspberry-PI Projekt dafür, wie man das nur mit einem Raspberry alleine bewerkstelligen soll, in meinem Versuch hatte es nicht geklappt) und 2 eigene NTPs basieren auf DCF-77. Die beiden Server mit der vermeintlich richtigen Zeit überstimmen den Server mit der richtigen GPS-Zeit.
Umgekehrt bei 2 GPS-Empfängern und Versuche diese zu manipulieren. Auch GPS-Signale lassen sich fälschen, wie dieses Video https://www.youtube.com/watch?v=VqLSKPYKeXU eindrucksvoll beweist.
Selbst wenn man nur 2 Server hätte, wird es dann gefährlich, wenn ein Angreifer sowohl GPS-, als auch DCF-77 Zeit falsch aussendet.
Um vor diesem Angriff wirklich sicher zu sein, sehe ich spontan 2 Maßnahmen.
1. Alle größeren Zeitsprünge in der Zeitquelle werden durch den NTP-Server verworfen, bis zum manuellen Eingriff wird lediglich die interne Uhr als NTP-Quelle verwendet
2. Zur Verhinderung einer zu langsamen oder schnellen externen GPS oder DCF-77 Empfangszeit: Implementation eines Vergleichsalogrithmus. Wie schnell laufen interne Referenzuhr und externe Uhr auseinander. Bei zu großer Differenz gibt es einen Alarm.
Genereller Schutz: Ein eigener vertrauenswürdiger NTP-Server an einem “geheimen” Ort, der via VPN abgefragt wird und bei zu großen Abweichungen in der Zeit wird ebenfalls Alarm geschlagen und man muss manuell der Sache nachgehen.
Hey. Thanks for your comment.
Indeed, you should rely on three (not two) independent time sources, such as GPS, Galileo, and DCF77. I have corrected the part in the blogpost that stated only two independent sources. Thanks for that.
Furthermore, I like your ideas about alarms when the external sources drift more than expected. I don’t know whether some NTP appliances such as the ones from Meinberg have implemented such features.
Cheers
Johannes
Hi Johannes,
I suggest you look at RFC 8633, https://datatracker.ietf.org/doc/rfc8633/. This covers “Network Time Protocol Best Current Practices.” The recommended minimum number of servers is four.
Hello Johannes,
Excellent post, thank you very much.
Even though I’m dealing with Internet infrastructure (network engineering for ISPs) and well aware of the importance of NTP and it’s use in the ddos landscape, I never gave much thought to the impact of messing with NTP as you mentioned in your post.
I wanted to ask if you ever witnessed attackers actually trying to utilise these techniques ?
Hi Amos,
there are a couple of attacking tools out there that leverage those NTP attack vectors. Hence it is likely that an advanced persistent threat uses those techniques in conjunction with other scenarios.
However, I by myself was not involved in any situations in which those attacks took place at the moment. (Or at least I was not aware of it. ;D)
My blogpost series is meant to rise the importance of NTP in general. Not only for mitigating the attacks but to increase the reliability at all.
Ciao
Johannes
Hi Johannes,
when using a Meinberg M300 with a TCXO, IMHO there is no use of redundant RF reception systems. Reliability and stability is a question of hardware – we cannot substitute a stable oscillator by software.
The M300 time generation is controlled by a high precision oscillator (let´s call it TCXO) with long latencies. The base-frequency of the TCXO *is only finally adjusted* by the GPS signal.
The signal of the TCXO (10,000000xxx MHz) in the M300 is sent by RF-cable to the Meinberg antenna. There it is multiplied and tunes the oscillator of the GPS-L1C – receiver inside the antenna to the GPS-frequency (1.575,42 MHz).
The received signal then returns through the cable at an intermediate-frequency of 35,4 MHz.
The deviation between the TCXO-signal and the received GPS-signal is used to slowly tweak the TCXO. This improvement-process takes hours.
This transmission of the low frequencies through the RF-cable enables to insert several 100 m of RF-cable between antenna and the NTP-Server.
If you spoof the GPS timecode slowly (attack), the TCXO will slightly shift the reception frequency and after a relevant deviation from the correct frequency, the reception of the GPS signal gets lost causing an alarm. Meanwhile the TCXO goes on without external correction (fallback).
If you spoil the GPS reception or the timecode at once (interference/ attack), there is the same fallback to the TCXO.
Meinberg offers oscillators of different precisions – between ± 4.3 msec per day (TCXO) up to ± 8.0 msec per year (Rb-Standard). This is without correction by GPS.
I understand the idea of keeping timeservers small and mostly software driven, but these constructions fail in case of severe interference (thunderstorm, LED-lights or mums shaver) or severe attacks. The most stable way is to use a clock-signal from a temperature / oven controlled quartz oscillator. This signal may be compared by software with the GPS- or a DCF77 signal to calculate a correction-factor.
In the 90ies, we used a small software that calculated a correction-factor for the PC-clock. This works fine and might lead to an interference stable design. It´s just a little more effort and cheap hardware.
So for redundancy of the time service one could think about a second (quartz related) backup system that is updated once a day, i.e. a good clock-module in the main server.
For a Meinberg NTP-Server with the special Meinberg GPS-Antenna you have to spend some bucks at eBay. On the other hand it doesn´t need any connection to the internet and it´s Stratum 1.
These Meinberg servers will also run without the Meinberg Antenna – but then you need a synchronization with other timeservers, IMHO once a day to once a month, depending on the stability of the oscillator.
Comment: Reproduction of the brilliant Meinberg antenna? In summary, I see no chance to build a reproduction of this very complex piece of RF-electronics.
There we need another approach.
Hi Johannes,
thank you for the great posts on NTP. Quite some insights.
We have been using the GPS based LeoNTP device by Leo Bodnar for quite some time now. It was extremely simple to set up, powered by PoE from the switch. Setting the IP-address with the front-panel knob it was up and running in 10 minutes.
This small box does provide basic status information via a front-panel display. There are no other monitoring capabilities besides this. All we needed.
http://www.leobodnar.com/shop/index.php?main_page=product_info&products_id=272
You might not like it as much, since it is IPv4 only ;-)
Fabian
Hey Fabian,
yep, thanks for the pointer to LeoNTP. Indeed it’s a cool small device. And I love PoE powered devices anyway. ;)
However, as you already mentioned, the lack of IPv6 makes it useful for me. ;(
By the way: Is there a WebGUI for configuration as well? Or a console? Can you update the firmware?
Cu
Johannes
No, port 123 only. One knob does it all. FW update is via USB.
It’s a ntp-toaster with a nice screen.
Hi Johannes, I recently came across your blog on NTP topics, with very good insights! Thanks for knowledge sharing. do you know if NTP attack statistics are publicly available on internet? thanks.
Hey Pascal.
Hm, good question. I don’t know. Haven’t searched for stats like that. I am sorry.
Thanks for answering.
I found NIST database interesting with search function :
https://nvd.nist.gov/vuln/search
Other than NTP-related risks, using external sources such as GPS, Galileo or DCF are also vulnerable to jamming and spoofing attacks.
I quite agree with your statement “You should not only rely on the time codes of a single foreign country”.
suggest you to have a look on our white paper
http://www.gorgy-timing.fr/FTP/COM/SCPTime-Traceable_and_Certified_Time_for_digital_sovereignty-V1.0.pdf
I strongly believe that standards should be maintained by some sort of impartly government. We all believe the kilo, kilometer or other measurements, not some good public effort or commercially ones (ntp.microsoft.com used by every Windows installation).
In Switzerland I use ntp.metas.ch as this is the proper time standard in Switzerland, government funded tax-money backed, just like good old KG and KM. I hope there are similar buraus of standards in other civilized countries.
Fun Fact: For testing purposes, I had a Raspberry Pi which was not able to access the Internet for its upstream NTP servers for almost 5 months. The drift was about 2300 seconds which is roughly 40 minutes. This was already enough to get some TLS/HTTPS certificate errors!