Aaron Toponce

So, we were just recently troubleshooting a connectivity issue at the office. Some things seemed slow, other things seemed fast. It was hard putting a finger on it. So, we starting pinging random stuff, trying to figure out exactly what was going on. Is it routing? Is in DNS? Is it layer 2? What gives?

Well, it turns out it was DNS. We were able to track it down, restart the DNS services, and everything was smoothing sailing after that. But, the interesting thing is, the GNU ping(1) utility was telling us this all along, and we ignored it. Consider the following ping(1) to this blog, and its relevant packet capture:

% ping -c 4 pthree.org              
PING pthree.org (166.70.136.38) 56(84) bytes of data.
64 bytes from tao.ae7.st (166.70.136.38): icmp_req=1 ttl=62 time=0.341 ms
64 bytes from tao.ae7.st (166.70.136.38): icmp_req=2 ttl=62 time=0.349 ms
64 bytes from tao.ae7.st (166.70.136.38): icmp_req=3 ttl=62 time=0.318 ms
64 bytes from tao.ae7.st (166.70.136.38): icmp_req=4 ttl=62 time=0.334 ms

--- pthree.org ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 0.318/0.335/0.349/0.021 ms

While in another terminal running tcpdump(1), we see the following output:

# tcpdump -ni eth0 port 53     
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
11:17:50.452323 IP 199.104.120.74.56928 > 198.60.22.2.53: 48462+ A? pthree.org. (28)
11:17:50.452925 IP 198.60.22.2.53 > 199.104.120.74.56928: 48462 1/5/21 A 166.70.136.38 (496)
11:17:50.453579 IP 199.104.120.74.43665 > 198.60.22.2.53: 46650+ PTR? 38.136.70.166.in-addr.arpa. (44)
11:17:50.454236 IP 198.60.22.2.53 > 199.104.120.74.43665: 46650 1/3/3 PTR tao.ae7.st. (181)
11:17:51.454981 IP 199.104.120.74.40485 > 198.60.22.2.53: 26170+ PTR? 38.136.70.166.in-addr.arpa. (44)
11:17:51.455430 IP 198.60.22.2.53 > 199.104.120.74.40485: 26170 1/3/3 PTR tao.ae7.st. (181)
11:17:52.455999 IP 199.104.120.74.51659 > 198.60.22.2.53: 31631+ PTR? 38.136.70.166.in-addr.arpa. (44)
11:17:52.456500 IP 198.60.22.2.53 > 199.104.120.74.51659: 31631 1/3/3 PTR tao.ae7.st. (181)
11:17:53.457183 IP 199.104.120.74.55632 > 198.60.22.2.53: 24544+ PTR? 38.136.70.166.in-addr.arpa. (44)
11:17:53.457632 IP 198.60.22.2.53 > 199.104.120.74.55632: 24544 1/3/3 PTR tao.ae7.st. (181)
^C
10 packets captured
11 packets received by filter
0 packets dropped by kernel

This output tells me that ping(1) does an initial forward lookup, then caches the resulting IP. Then, FOR EACH ICMP PACKET SENT, it does a reverse lookup on the IP. This is actually shown in the output, if you pay attention. I wasn't paying attention. Of course, according to the ping(1) manpage, you can pass the "-n" switch to prevent ping(1) from doing reverse lookups.

At any event, I'm not sure if you were aware of this or not, but I usually get so caught up in latencies and paths, that I'm not always paying attention to DNS. I guess ping(1) can be a great diagnostic tool for DNS as well. Learn something new each day.