Network Troubleshooting: ping, mtr, tcpdump, ss and netem
A layer-by-layer troubleshooting method on a lab network: simulate latency and loss with tc netem, find MTU problems, read SYN retries, probe endpoints.
When something "cannot connect", the fastest engineers do not guess. They ask the same few questions in the same order, from the bottom of the stack up, and they use a tool that answers each question with evidence: an address, a route, a packet, a timer. Most network problems end at the first question that gets a surprising answer.
In this part you get that order as a checklist, then practise each step on the lab. You make the network bad on purpose with tc netem (latency, jitter and loss), find an MTU problem, watch TCP retry a dropped connection in a packet capture, and finish with monitoring that notices problems before users do.
Prerequisites
- A fresh lab from Part 1: run
sudo ./netlab.sh down && sudo ./netlab.sh upbefore you start, because every part changes the lab. - The errors from Part 1's When It Fails, Read the Error section.
A Troubleshooting Order
Work up the layers, and stop at the first step that fails:
| Step | Question | Command | A surprising answer means |
|---|---|---|---|
| 1 | Does the name resolve to the right address? | getent hosts db, dig +short db.shop.internal |
DNS (Part 3) |
| 2 | Is there a route, and which way does it go? | ip route get 10.0.3.10, tracepath -n 10.0.3.10 |
Routing, VPN, cloud route tables (Parts 2 and 7) |
| 3 | Does the port answer? Refused or silent? | nc -zv -w 3 db 5432 on the client, ss -tlnp on the server |
Service down, wrong bind address, firewall (Parts 1 and 4) |
| 4 | Does TLS complete with the right certificate for this name? | openssl s_client -connect host:443 -servername host -verify_hostname host -verify_return_error -brief |
Expired or wrong certificate, missing CA |
| 5 | Does the application answer correctly, and how fast? | curl -v, curl -w timings, the service's logs |
The app, a proxy, a dependency |
Run the check from the same place as the failing program: the same host, container or pod. A database that answers from your laptop proves nothing about the pod that cannot reach it.
Latency, Jitter and Loss
ping measures the round trip and loss to one host; mtr combines ping and traceroute and measures every hop on the path. First the healthy lab, then the same path with a bad link. tc attaches a queueing discipline to an interface; netem (network emulator) delays, varies and drops packets on their way out. On the router, apply it to the port that leads to client:
How to read it:
pingsummarises the bad link: an average round trip near 40 ms and a largemdev(the standard deviation), which is the jitter. netem drops 5% of packets at random, so a run of 50 pings can lose none or several; themtrrun below is a second sample of 50. Jitter hurts voice and video more than a steady delay does.mtrlists every hop, and this run shows its classic trap. The last line,10.0.2.11, has the real numbers: a few percent loss and about 40 ms. The router's line shows far more "loss" than the destination behind it, which cannot be real loss, because every packet to10.0.2.11passed through that router. Linux limits the ICMP "time exceeded" replies thatmtrrelies on to about one per second per destination (net.ipv4.icmp_ratelimit), andmtr -i 0.2asks five times a second. Judge a path by its last line, and treat loss at a middle hop that does not continue to the end as rate limiting.netemworked in one direction only, on the router's way out toclient. Real problems are often one-way too;mtrfrom both ends tells them apart.
This is also the easiest way to test how your own application behaves on a bad network: run it behind tc netem and watch timeouts, retries and the user experience. The posts simulating a slow internet connection and network tools that simulate slow connections cover the options on macOS and Windows too.
What Loss Does to TCP
TCP hides loss by sending again, and you pay for it in speed. iperf3 measures throughput between two machines. Start its server on web1, compare the two directions of the bad link, then look at the kernel's counters:
The column Retr counts retransmitted segments on the sending side, and nstat shows the same kernel counters (since the namespace started) on web1, which sent through the lossy link. Through the lossy link, TCP backs off its congestion window after every loss, and throughput collapses by orders of magnitude, although only 5% of packets were lost. That is why a little packet loss feels like a dead network to users, and why nstat's TcpRetransSegs (or node_netstat_Tcp_RetransSegs in Prometheus) is one of the most useful counters on a server. The veth links of the lab are not real network cards, so the absolute numbers mean little; the gap between the two directions is the lesson.
A Packet Capture of a Dropped Connection
A firewall that drops packets produces silence (Part 4), and a capture shows what the silence looks like. Make db drop PostgreSQL connections from web1:
Then capture on web1 while it tries to connect, with no timeout of its own (this takes over two minutes). -ttt prints the time since the previous packet:
The client sent the same SYN (same source port) eleven times. On current kernels (Linux 6.5 and later, so Ubuntu 24.04's and 26.04's too) the first few retries come at a fixed one-second interval (tcp_syn_linear_timeouts), and after that the wait doubles each time (about 2, 4, 8, 16 and 33 seconds here), TCP's exponential backoff. After the last SYN the kernel waits once more, and gives up after more than two minutes in total. Any program that does not set its own connect timeout waits that long for every attempt to an address that drops packets. Set explicit connect timeouts in every network client you write or configure.
To save a capture for Wireshark, write it to a file with -w and read it back with -r; keep captures short and filtered, because they contain everything on the wire, including credentials in plain-text protocols:
sudo ip netns exec lb tcpdump -nn -i eth0 -c 200 -w /tmp/lb.pcap 'tcp port 80'
tcpdump -nn -r /tmp/lb.pcap | head
MTU Problems
Every link has a Maximum Transmission Unit, the largest packet it carries (1500 bytes on Ethernet). Tunnels and overlays add headers and lower it: WireGuard to 1420, VXLAN to 1450, some VPNs and PPPoE links lower still. A path with a smaller MTU somewhere in the middle relies on ICMP "fragmentation needed" messages to tell senders to use smaller packets; when a firewall drops those messages, small requests work and large responses hang forever. Give the data subnet a smaller MTU and find it:
tracepath reports the smaller pmtu at the hop where it starts. ping -M do sets the "don't fragment" flag: once the host knows the path MTU, a packet that is too large fails locally with message too long, and one that fits gets through. When you find an MTU problem, either let ICMP "fragmentation needed" (ICMPv6 "packet too big" for IPv6) through every firewall on the path, or lower the MTU (or the TCP MSS) at the tunnel.
Where Does the Time Go?
A slow request can be slow in DNS, in the TCP or TLS handshake, or in the server. curl's timers split the total. Save the format once:
dns %{time_namelookup}s\n
connect %{time_connect}s\n
tls %{time_appconnect}s\n
first byte %{time_starttransfer}s\n
total %{time_total}s\n
Each timer is cumulative, measured from the start. In the slow request DNS, connect and TLS take the same few milliseconds as before, and the jump is between TLS and the first byte: the server (here, the /slow endpoint, which waits two seconds) is slow, not the network. Large gaps in the other places point elsewhere: dns to the resolver, connect to latency or a busy server's accept queue, tls to the handshake or certificate checks such as OCSP.
Sockets and Connection States
ss shows every socket and its state. A summary, and a count by state, are the first things to look at on a busy server:
LISTEN: servers waiting for connections.ss -tlnpshows them with their process.ESTAB: open connections.TIME-WAIT: connections that this side closed first, kept for about 60 seconds so that late packets cannot confuse a new connection with the same 4-tuple. Thousands of them on a client mean it opens a new connection per request; use keep-alive and pooling (Part 5 usedkeepalivetowards the app servers for this).SYN-RECVpiling up on a server means half-open connections: a SYN flood, or clients that cannot complete the handshake.CLOSE-WAITpiling up means the application received the peer's close and never closed its own socket: a bug in the application, not the network.
Monitor Before Users Notice
Interactive tools answer "what is wrong now". Monitoring answers "is anything wrong" all the time. Two kinds of checks cover a network:
- Black-box probes from outside: can a client resolve the name, connect, complete TLS and get a 200, and how long does it take? The Prometheus blackbox exporter does exactly this for HTTP, TCP, DNS and ICMP.
- White-box metrics from inside: interface errors and drops, retransmissions, connection counts, conntrack table use. The Prometheus node exporter exposes these from
/procand/sys.
Run the blackbox exporter on client, the outside view. Download it from its releases page (check the SHA-256 sum), and give it a config with three probe modules:
modules:
https_2xx:
prober: http
timeout: 5s
http:
preferred_ip_protocol: ip4
tls_config:
ca_file: /etc/netlab/tls/ca.crt
dns_shop:
prober: dns
timeout: 5s
dns:
query_name: www.shop.example
query_type: A
preferred_ip_protocol: ip4
valid_rcodes: [NOERROR]
tcp_connect:
prober: tcp
timeout: 5s
tcp:
preferred_ip_protocol: ip4
Each probe returns metrics; Prometheus scrapes the /probe URL on a schedule (the target goes in the URL, which takes a relabelling step in the scrape config) and alerts on them. probe_success is 1 or 0, probe_duration_seconds is the whole check, and probe_ssl_earliest_cert_expiry is the Unix time when the first certificate in the chain expires, which turns certificate renewals into an alert instead of an outage. Rules that the guide's companion repository checks with promtool:
groups:
- name: network
rules:
- alert: EndpointDown
expr: probe_success == 0
for: 2m
labels:
severity: page
annotations:
summary: '{{ $labels.instance }} has failed its probe for 2 minutes'
- alert: EndpointSlow
expr: probe_duration_seconds > 1
for: 10m
labels:
severity: ticket
annotations:
summary: '{{ $labels.instance }} takes {{ $value | humanizeDuration }} to answer'
- alert: CertificateExpiresSoon
expr: probe_ssl_earliest_cert_expiry - time() < 14 * 86400
for: 1h
labels:
severity: ticket
annotations:
summary: 'The certificate of {{ $labels.instance }} expires in less than 14 days'
- alert: TcpRetransmitsHigh
expr: rate(node_netstat_Tcp_RetransSegs[5m]) / rate(node_netstat_Tcp_OutSegs[5m]) > 0.02
for: 15m
labels:
severity: ticket
annotations:
summary: 'More than 2% of TCP segments from {{ $labels.instance }} are retransmissions'
Page a person only for what users feel (an endpoint down for minutes), and send the rest to a ticket queue. The page, ticket or ignore post goes through that triage.
A Small Check Script
Not everything needs Prometheus. A script that checks the paths a service depends on, with timeouts and a clear exit status, is useful in a deploy pipeline, a cron job or right after a firewall change. Save it as netcheck.sh and make it executable (chmod +x netcheck.sh):
#!/usr/bin/env bash
# netcheck.sh HOST:PORT...: check that each name resolves and that one of its addresses
# accepts TCP connections on the port
set -u
if [ "$#" -eq 0 ]; then
echo "usage: $0 HOST:PORT..." >&2
exit 2
fi
status=0
for target in "$@"; do
host=${target%:*}
port=${target##*:}
case "$port" in
'' | *[!0-9]*)
printf 'FAIL %-22s not a HOST:PORT pair\n' "$target"
status=1
continue
;;
esac
# every address of the name, in the resolver's order; the lookup itself is limited to 3 s
addrs=$(timeout 3 getent ahosts "$host" | awk '!seen[$1]++ { print $1 }')
if [ -z "$addrs" ]; then
printf 'FAIL %-22s name does not resolve\n' "$target"
status=1
continue
fi
result=""
for addr in $addrs; do
start=$(date +%s%N)
if out=$(nc -n -zv -w 3 "$addr" "$port" 2>&1); then
result=$(printf 'ok %-22s %s, %d ms' "$target" "$addr" $((($(date +%s%N) - start) / 1000000)))
break
fi
case "$out" in
*"timed out"*) reason="no answer within 3 s" ;;
*": "*) reason=${out##*: } ;;
*) reason="connection failed" ;;
esac
result=$(printf 'FAIL %-22s %s, %s' "$target" "$addr" "$reason")
done
echo "$result"
case "$result" in FAIL*) status=1 ;; esac
done
exit "$status"
Three ways to fail, three different messages: a refused port, a name that does not exist, and (if you add a firewall rule) a timeout. The exit status of 1 makes a pipeline step or a cron wrapper fail loudly.
Common Mistakes
- Testing from the wrong place. Run checks from the failing host, container or pod, with its DNS settings and its firewall.
- Concluding "the network is fine" from ping. ICMP can pass while a firewall drops the TCP port, and the other way round.
- Reading loss at a middle
mtrhop as a fault. Look at the final hop. - Watching averages. A 50 ms average can hide a p99 of 2 seconds. Alert on percentiles and on failures.
- No timeouts in clients. Without a connect timeout, a dropped SYN takes over two minutes to fail.
- Capturing everything. Filter captures by host and port, keep them short, and treat pcap files as secrets.
Summary
- Troubleshoot in order: name, route, port, TLS, application, and run each check from where the failure happens.
tc netemsimulates delay, jitter and loss (and, with itsrateoption, a slow link); a few percent of loss can cut TCP throughput dramatically.tcpdump -tttshows SYN retries: one second apart at first, then doubling, for over two minutes;tracepathandping -M dofind MTU problems.- curl's timers split a slow request into DNS, connect, TLS and server time;
ssstates point to pooling problems, floods and application bugs. - Black-box probes (blackbox exporter) and white-box counters (node exporter) catch failures, slowness and expiring certificates before users do.
Next, Part 9 puts the network itself into code: firewall rules, cloud networks and tests in a pipeline.
Keep Learning
- Ping: request timed out vs destination host unreachable
- Capturing mobile phone traffic with Wireshark
- Simulators: latency percentiles, PromQL playground
- Monitoring and logging in the Kubernetes guide
Tags
Found an issue?