If you’re running your own DNS resolver you’re probably interested in some benchmark tests against it, such as: how fast does my own server (read: Raspberry Pi) answer to common DNS queries compared to 8.8.8.8.

In this blogpost I am showing how to use two tools for testing/benchmarking DNS resolvers: namebench & dnseval. I am listing the defaults, giving some hints about them and showing examples in which I tested some private and public DNS resolvers: a Fritzbox router, a Raspberry Pi with Unbound, Quad9, OpenDNS, and Google Public DNS.

namebench

Namebench is an open-source DNS benchmark utility listed at Google Code Archive. You can install it on any Ubuntu-like Linux with sudo apt-get install namebench. The config files are at  /etc/namebench/ while certain lists that are used by the program are here: /usr/share/namebench/data/. It ships with a GUI but I prefer the usage through the CLI. Be careful: Without any options it immediately starts testing some (many!) servers. For information about its options you must use the -h switch: namebench -h.

Namebench has a couple of input sources while the most interesting ones are “alexa“, “cachehit” and “cachemiss“. While the first one queries different FQDNs out of a built-in Alexa list the other two options test exactly what the names imply: either cache hits (by only querying the same name again and again) or cache misses by querying random names for which the DNS resolver must first look up the record itself before delivering it to the client.

Note that the “cache-hit.txt” file which is used for the cachehit procedure queries the A record of “a.root-servers.net.”. In my test cases (below) this made a difference compared to some other DNS queries. I am not quite sure why but I suppose that this is related to some built-in lists for the root servers into the DNS resolvers. For example, Unbound has a “root.hints” file which holds the A and AAAA records for them. Hence, this file is queried rather than the cache itself. In my examples I changed the content of the file which changed the test results dramatically as well!

Furthermore note that namebench does some kind of sanity checks which are completely outdated. I commented out any sanity lines within the sudo nano hostname_reference.cfg file to get rid of the warnings in the results. These are not correct anymore since the offered service such as Twitter or PayPal use other A records nowadays.

A sample command of namebench looks like that which uses the alexa input source and tests the Google Public DNS server:

The results are stored at /tmp/ such as /tmp/namebench_2017-10-27_1204.html.

(There is a german article on Heise.de concering namebench.)

dnseval

Dnseval is one out of three tools from the DNSDiag suite by Babak Farrokhi. At GitHub there are some install instructions. I already published a few information about dnseval here: Compare & Troubleshoot DNS Servers: dnseval. However, I merely looked at troubleshooting issues with DNS flags, TTL, and caching issues. Today I want to use dnseval to benchmark the DNS resolvers which is almost the same though. ;) I am simply using much more queries (such as -c 2000) while I started the measurement after I initially queried the name from all servers in order to have it in their caches. Be aware that dnseval only tests cache hits since it queries the same name again and again.

A sample run to test the resolver found at /etc/resolv.conf is:

Test Examples & Results

For the following examples I queried and compared the following five DNS resolvers:

1. My LAN router, AVM Fritzbox 7430 with FRITZ!OS 06.83 (DNS forward to my ISP Deutsche Telekom) at 2003:50:aa0c:3e00:ca0e:14ff:fe7e:339f
2. A Raspberry Pi 1 B with Unbound version 1.4.17 (recursive DNS resolver that generates iterative queries throughout the DNS tree) at 2003:50:aa0c:3e00:ba27:ebff:fec9:1637
3. OpenDNS at 2620:0:ccc::2
4. Google Public DNS at 2001:4860:4860::8888
5. Quad9 at 2620:fe::fe

For every run I queried 2000 entries.

namebench alexa

At first I used the alexa input source:

The winner was OpenDNS with a mean response of 75 ms (green) followed by the local Fritzbox with 87 ms (purple). Unbound (151 ms, blue) and Google Public DNS (178 ms, orange) are the last ones. I expected that Google would be faster. Hm. On the graph you can see that Unbound (the only iterative working DNS resolver in the test case) has a slow-growing curve since it really must query each name independently since I don’t have many cache hits in my local network compared to all public resolvers.

namebench cache-hit

Now the built-in “cachehit” test:

Huge differences between my two local DNS resolvers: the Fritzbox router (first line, average time of  1.8 ms!) was much faster than Unbound (last line, 9.5 ms). I was kind of shocked. I did not expect that the Raspberry Pi with Unbound was that much slower than a simple router! BUT:

namebench cache-hit UPDATED!

I changed the “cache-hit.txt” file (as already explained above) to query “weberblog.net.” rather than “a.root-servers.net.”. Here are the results:

Ah, here we go. Now the Fritzbox (first line, 2.17 ms) is only a little bit faster than Unbound (second line, 2.43 ms). As already noted I suppose that this test case now was a real cache-hit while the built-in version from namebench did query some local root.hints files. In any case, the three public DNS resolvers are almost comparable while (of course) slower due to the network patch.

namebench cache-miss

The last run of namebench, this time with some randomly queried names:

The winner again was the local Fritzbox router (purple, 14.63 ms) followed by OpenDNS (green, 15.52 ms), Unbound (blue, 16.79), and Quad9 (turquoise, 19.99 ms). Google Public DNS again was the slowest one (orange, 28.20 ms). As you can see in the graph, almost all curves are the same since all DNS server must query the random value one by one. The orange line from Google is shifted to the right, probably due to a slower routing path.

dnseval (cache-hit)

This is the cache hit test from dnseval. The two local DNS resolvers on my network are quite faster than the remote ones. The Unbound resolver on my Raspberry Pi had an average of 2.4 ms while my Fritzbox router had 1.8 ms:

Compared to the updated cache-hit test from namebench the order of the DNS resolvers is almost the same: Fritzbox won, followed by Unbound which are both faster than the three public resolvers. Looking at them, OpenDNS was the fastest, but this time followed by Google Public DNS while Quad9 was the slowest.

Conclusion

It’s not easy to interpret the measured values in the correct way. You must precisely know what you are testing. As already pointed out you can test cache-hits, cache-misses or “real world” queries such as Alexa Top 1M which will have many cache-hits on public resolvers while just a few on your own resolvers.

In any case you are NOT benchmarking the load of a DNS resolver such as “how many DNS queries can it handle in parallel” or “how many clients can be handled by a Raspberry Pi” but the latency for single DNS queries. That is: you’re testing the IP routing network latency + the mere DNS server latency together.

[UPDATE] Traceroutes & Chaos Queries

As Bill requested in the comment section, here are the traceroutes and chaos query results from the three public DNS servers. First the traceroutes. I used mtr to display the routing paths as well:

Note that only Quad9 responded to the chaos query:

Featured image “Koblenz – Hochwassermarken am Kranhaus” by onnola is licensed under CC BY-SA 2.0.