Diagnostic Tools #

Diagnosing network and web server problems shouldn’t be done based on baseless guessing or intuition. Mastering the right diagnostic tools can turn a confusing troubleshooting process into a systematic, precise, and efficient investigation. This article reviews in depth the arsenal of mandatory helper tools for every system administrator managing Caddy, complete with production command parameters and detailed result interpretation.

Caddy Log & Metric Observability Data Flow #

When managing Caddy at scale, you don’t just rely on manual terminal checks, but compose an integrated data pipeline to monitor server health visually.

Here’s a visualization of the log and metric data flow from Caddy to a centralized monitoring system:

flowchart LR
    Caddy["Caddy Web Server"] -->|"JSON Logs"| LogFile["api-access.log"]
    LogFile -->|"Scrape"| Promtail["Grafana Promtail"]
    Promtail -->|"Push HTTP"| Loki["Grafana Loki"]
    Loki -->|"Query LogQL"| Grafana["Grafana Dashboard"]
    
    Caddy -->|"Metrics (/metrics)"| Prometheus["Prometheus Server"]
    Prometheus -->|"Query PromQL"| Grafana
    
    style Caddy stroke:#0288d1,stroke-width:2px
    style Grafana stroke:#7b1fa2,stroke-width:2px

By understanding this flow, you can trace where your diagnostic data is stored and how to leverage it during system failures.

curl — The HTTP Swiss Army Knife #

curl is the main tool for verifying HTTP responses directly from the terminal. You can isolate header behavior, status codes, and map latency times at every connection stage.

Explaining curl Timing Parameters #

When you run latency tests with curl, you get several time variables that are very important to analyze:

  1. time_namelookup (DNS Lookup): The time (in seconds) from the request start until domain name resolution finishes. If this value is high (e.g., > 0.5s), your DNS server is slow or experiencing propagation problems.
  2. time_connect (TCP Handshake): The time needed to establish the TCP connection (3-way handshake) between the client and Caddy. A large value indicates physical network latency or a firewall slowing down SYN-ACK packets.
  3. time_appconnect (TLS/SSL Handshake): The time until the TLS handshake finishes. If this stage takes long, it could be heavy SSL computation on the server or suboptimal protocol version negotiation.
  4. time_starttransfer (TTFB - Time to First Byte): The time from when the request is sent until the first response byte is received from Caddy. This is a backend performance indicator: if TTFB is high, the backend is slow at processing application logic or database queries.
  5. time_total (Total Time): The overall transaction completion time.
# 1. Send a request with complete verbose output
# Shows TLS handshake details, request headers (>), and response headers (<)
curl -v https://example.com

# 2. Fetch only the response headers (without downloading the file body)
curl -I https://example.com

# 3. Display only the HTTP status code (very useful for automation scripts)
curl -o /dev/null -s -w "%{http_code}\n" https://example.com

# 4. Detailed connection time metric breakdown
curl -o /dev/null -s -w "
DNS Lookup:    %{time_namelookup}s
TCP Handshake: %{time_connect}s
TLS Handshake: %{time_appconnect}s
Backend TTFB:  %{time_starttransfer}s
Total Time:    %{time_total}s
Response Size: %{size_download} bytes
HTTP Status:   %{http_code}
" https://example.com

# 5. Send a request with custom headers and token authentication
curl -H "Authorization: Bearer ***" \
     -H "X-Gateway-Trace: debug-mode" \
     https://api.example.com/v1/users

# 6. Send JSON-format data (POST Request)
curl -X POST https://api.example.com/v1/users \
     -H "Content-Type: application/json" \
     -d '{"name": "Budi", "role": "admin"}'

# 7. Follow the redirect chain (Location header) and see the flow
curl -L -v https://www.example.com 2>&1 | grep -E "< HTTP|Location:"

# 8. Force testing using a certain TLS version
curl --tlsv1.3 https://example.com
curl --tlsv1.2 https://example.com

# 9. Ignore SSL certificate validation (For local testing purposes only!)
curl -k https://localhost:8443/

openssl — TLS/SSL Diagnostics #

When Caddy experiences SSL/TLS certificate problems, you use openssl to deeply examine the certificate chain and cipher suite algorithm compatibility.

# 1. Download and display all certificate data served by Caddy
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | \
    openssl x509 -noout -text

# 2. Display the subject, issuer, and validity period summary
echo | openssl s_client -connect example.com:443 2>/dev/null | \
    openssl x509 -noout -subject -issuer -dates

# 3. Display the entire certificate trust chain (Chain of Trust)
# Helps ensure the Root CA and Intermediate CA are sent intact
echo | openssl s_client -connect example.com:443 -showcerts 2>/dev/null | \
    grep -E "subject=|issuer="

# 4. Verify a local certificate file using the system CA file
openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/example.com/example.com.crt

# 5. Check the certificate expiration date
echo | openssl s_client -connect example.com:443 2>/dev/null | \
    openssl x509 -noout -enddate

dig / nslookup — DNS Diagnostics #

Caddy’s automatic certificate challenges (ACME) depend heavily on DNS. If your domain doesn’t correctly point to the server’s public IP, Let’s Encrypt fails validation.

# 1. Fetch the IP Address (A record) briefly
dig example.com A +short

# 2. Fetch the IPv6 IP Address (AAAA record)
dig example.com AAAA +short

# 3. Trace the DNS resolution path from the Root DNS to your domain's nameserver (DNS Trace)
# Very useful for detecting stale DNS caching
dig example.com +trace

# 4. Check TXT records
# Mandatory when debugging DNS-01 Challenge failures
dig _acme-challenge.example.com TXT

# 5. Test DNS propagation to various world public DNS resolvers quickly
for dns in 8.8.8.8 1.1.1.1 9.9.9.9; do
    echo -n "DNS $dns: "
    dig @$dns example.com A +short
done

ss / netstat — Network Sockets #

You use ss to detect which ports are open, detect port binding conflicts, and count the number of active connections being handled by the Caddy process.

# 1. Display all TCP ports in LISTENING status complete with their process names
sudo ss -tlnp

# 2. Filter the specific ports used by Caddy and the Admin API
sudo ss -tlnp | grep -E ':80|:443|:2019'

# 3. Count the number of active TCP connections currently connected to the Caddy process
sudo ss -tnp | grep caddy | wc -l

# 4. Analyze the connection count by socket status (ESTABLISHED, TIME_WAIT, etc.)
sudo ss -tan | awk '{print $1}' | sort | uniq -c | sort -rn

journalctl — Systemd Logs #

On modern Linux distributions, Caddy’s standard logs when run as a system service are collected by the systemd-journald unit.

# 1. Monitor Caddy logs in real time (Tailing)
sudo journalctl -u caddy -f

# 2. View Caddy logs specifically for the current system boot
sudo journalctl -u caddy -b

# 3. Filter Caddy logs within the last 1 hour
sudo journalctl -u caddy --since "1 hour ago"

# 4. Filter logs specifically at the ERROR and WARNING levels only
sudo journalctl -u caddy -p err..warning --no-pager

# 5. Extract and search for certain text in the logs (e.g., TLS handshake issues)
sudo journalctl -u caddy | grep -i "handshake" | tail -n 20

tcpdump — Raw Packet Capture #

When the network problem is at the lowest packet level (like MTU route disruptions or packet cutting by network firewalls), you need tcpdump to record raw network packets.

# 1. Record traffic on port 443 on a certain network interface (e.g., eth0)
# -n disables domain name resolution for fast processing
sudo tcpdump -i eth0 port 443 -n

# 2. Record port 80/443 data packets and save them to a pcap file
# This file can later be downloaded and visually analyzed using Wireshark
sudo tcpdump -i any port 80 or port 443 -w /tmp/caddy-traffic.pcap

# 3. Read the pcap recording file contents in a limited way
sudo tcpdump -r /tmp/caddy-traffic.pcap | head -n 30

Analyzing PCAP Capture Files with Wireshark #

After creating the caddy-traffic.pcap file using tcpdump, you can open it in Wireshark for advanced debugging:

  • Finding TCP Retransmissions: Filter with tcp.analysis.retransmission. If there are many retransmission packets, there’s packet loss between the client and Caddy.
  • Analyzing TLS Client Hello: Find Client Hello packets to see the TLS version and cipher suites offered by the client browser using the tls.handshake.type == 1 filter.
  • Identifying RST (Reset) Packets: Filter with tcp.flags.reset == 1 to see whether the connection was closed unilaterally by the client, Caddy, or a network middleware.

Caddy Admin API Diagnostics #

Caddy provides a built-in REST API on the localhost:2019 port for directly monitoring the internal runtime state.

Explaining the Upstream API JSON Response #

When you call the http://localhost:2019/reverse_proxy/upstreams/ endpoint, Caddy returns an array of JSON objects representing the status of each upstream server:

  • address: The backend IP and port address (e.g., 127.0.0.1:3000).
  • healthy: The upstream health boolean status based on passive/active monitoring.
  • num_requests: The number of active HTTP connections currently being processed by that upstream.
  • fails: The accumulated TCP connection failures detected within the passive monitoring time window.
# 1. Actively monitor backend (upstream) server health status
curl -s http://localhost:2019/reverse_proxy/upstreams/ | \
    jq -r '.[] | "\(.address): \(if .healthy then "UP" else "DOWN" end) (active connections: \(.num_requests), fails: \(.fails))"'

# 2. Display the active routing configuration translated by Caddy
curl -s http://localhost:2019/config/apps/http/servers/srv0/routes/ | jq .

wrk / ab — Load Testing #

Before releasing a new Caddy configuration to the production environment, you need to test the server’s endurance under high load (load testing).

# Using wrk (a high-performance HTTP-based load testing tool)
# Parameters: 4 threads, 100 simultaneous connections, 30-second duration
wrk -t4 -c100 -d30s https://example.com/

# Using Apache Benchmark (ab) for quick testing
# Sends a total of 1000 requests with 100 simultaneous request concurrency
ab -n 1000 -c 100 https://example.com/

mtr — Network Path Analysis #

mtr combines ping and traceroute functionality to analyze the network connection quality from your server to the destination server/client.

# Run interactive network route monitoring
mtr example.com

# Generate a static report (non-interactive, sending 20 packets)
# Useful for sending to cloud infrastructure teams
mtr --report --report-cycles 20 example.com

httpie — A More User-Friendly curl Alternative #

httpie is an alternative terminal tool for sending HTTP requests with a much more human-rememberable syntax and color text visualization (syntax highlighting). This tool is highly recommended for API developers because you no longer need to think about double-quote wrapping and complicated Content-Type header definitions when sending JSON-format data, since httpie automatically parses input arguments and adjusts the right request headers.

# 1. Send a regular GET request (automatically beautifully formatted output)
http https://api.example.com/v1/users

# 2. Send JSON-format POST data (no need to write manual headers)
http POST https://api.example.com/v1/users \
    name="Dewi" \
    email="[email protected]"

# 3. Send a request with basic authentication
http --auth admin:secret https://example.com/admin/

Grafana + Loki + Promtail Stack (Centralized Log Visualization) #

For long-term log analysis, you aggregate Caddy’s JSON logs centrally using a Docker compose stack.

The docker-compose.yml file for running a local Loki stack:

version: "3.8"

services:
  loki:
    image: grafana/loki:3.0.0
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - monitoring

  promtail:
    image: grafana/promtail:3.0.0
    volumes:
      - /var/log/caddy:/var/log/caddy:ro
      - ./promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:10.4.0
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

The Promtail configuration file (promtail-config.yml) for parsing Caddy’s JSON log files:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: caddy-access-logs
    static_configs:
      - targets:
          - localhost
        labels:
          job: caddy
          __path__: /var/log/caddy/*.log
    pipeline_stages:
      - json:
          expressions:
            status: status
            duration: duration
            method: request.method
            uri: request.uri
            client_ip: request.remote_ip
      - labels:
          status:
          method:
          client_ip:

With this monitoring infrastructure, you can create HTTP status code (2xx, 4xx, 5xx) bar chart graphs and monitor latency spikes graphically from the Grafana dashboard.

Summary #

  • curl Time Analysis — Use the formatting parameter metrics on curl to isolate whether high latency comes from DNS resolution, TCP handshakes, or the backend.
  • SSL Trust Chain — Verify the Intermediate and Root CA delivery integrity using the openssl s_client -showcerts utility to prevent untrusted SSL issues on client devices.
  • DNS Route Tracing — Use the dig +trace command to detect wrong DNS record propagation and caching when debugging ACME challenge failures.
  • Socket Connections — Diagnose port availability and socket binding conflicts on Linux systems instantly using the ss -tlnp utility.
  • Packet Sniffing — Leverage tcpdump to record raw network packets into pcap files for more detailed analysis using the external Wireshark application.
  • Loki & Grafana Visualization — Integrating Caddy’s structured JSON logs with Loki and Promtail gives full visibility into long-term API performance.

← Previous: Caddy Validate   Next: Best Practices →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact