Debugging Configuration #

Debugging a Caddy configuration that doesn’t behave as expected can be a confusing process if you don’t know where to look for information. Unlike traditional web servers that often hide their internal processing details, Caddy has a very rich logging engine and a REST-based Admin API allowing you to see the active configuration memory contents in real time. This article presents a systematic guide for debugging various configuration problems — from non-matching routes and empty placeholders to TLS handshake failures.

HTTP Request Lifecycle & Debug Interception Points #

When debugging, you must understand how Caddy processes requests from the moment the TCP connection is formed until the response is sent back to the client. The diagram below shows Caddy’s internal processing pipeline (HTTP pipeline) flow along with the points where you can insert debug instruments:

flowchart TD
    TCPConnection["TCP Connection Opened"] --> TLSHandshake{"TLS Handshake?"}
    
    TLSHandshake -- Failed --> TLSFailed["Handshake Failed (tls: no certificate)"]
    TLSHandshake -- Success --> TLSSuccess["Handshake Successful (Valid Certificate)"]
    
    TLSSuccess --> HTTPMatches{"Named Matcher Matching?"}
    
    HTTPMatches -- No Match --> NoMatch["Use the Default Handler / Fallback 404"]
    HTTPMatches -- Match --> Match["Execute the Middleware Chain"]
    
    subgraph MiddlewarePipeline["Caddy Middleware Pipeline"]
        Log["Log Request ID & Headers (X-Debug-Route)"] --> Rewrite["Rewrite URL / Redirect (redir / rewrite)"]
        Rewrite --> Auth["Authentication (basicauth / JWT)"]
        Auth --> Route["Reverse Proxy to Upstream / File Server"]
    end
    
    Match --> MiddlewarePipeline
    Route --> Response["Downstream Response (Header Transformation)"]
    
    style TCPConnection stroke:#0288d1,stroke-width:2px
    style MiddlewarePipeline stroke:#7b1fa2,stroke-width:2px

Systematic Debugging Principles #

So the problem-finding process runs efficiently, you should get used to applying these four troubleshooting pillars:

  1. Reproduce: Make sure you can trigger the error consistently. Record the exact conditions (URL path, HTTP headers, request method, and client device type) when the error occurs.
  2. Isolate: Separate each component to find the error location. Is the problem in DNS resolution, the Caddy-to-backend network, the Caddyfile configuration, or a bug in your own backend code? Simplify the configuration to the minimal point that can still reproduce the error.
  3. Analyze: Leverage system logs, JSON configuration adaptation, and the Admin API to monitor the server’s internal state.
  4. Fix & Verify: Apply the solution, run syntax validation before deploying, and test again using automated scripts to make sure there’s no performance regression.

Debug Mode #

The first and most important step when finding anomalies in Caddy is enabling Debug Mode. By default, Caddy only logs at the INFO level and above to keep log file sizes small and save disk I/O. By raising the level to DEBUG, Caddy logs the details of every route matching decision, TLS handshake header, and upstream dial attempt.

1. Enabling Debug Mode in the Caddyfile #

Add the debug global option at the very top of your Caddyfile:

{
    # Enable debug level logging globally
    # WARNING: The logs will be very verbose. Turn it back off after the debug process finishes!
    debug
}

example.com {
    reverse_proxy localhost:3000
}

After saving the Caddyfile, reload the configuration:

sudo systemctl reload caddy

2. Enabling Debug Mode Dynamically via the Admin API (Without Restart) #

If you don’t want to change the Caddyfile or can’t restart an active production environment, you can raise the log level to debug instantly through the Admin API:

# Change the default log level to debug dynamically
curl -X PUT http://localhost:2019/config/logging/logs/default/level \
    -H "Content-Type: application/json" \
    -d '"debug"'

# Monitor debug logs in real time from the terminal
sudo journalctl -u caddy -f | grep -i "debug"

# Return to the info level after finishing the tracking
curl -X PUT http://localhost:2019/config/logging/logs/default/level \
    -H "Content-Type: application/json" \
    -d '"info"'

Reading the Active Configuration via the Admin API #

The Caddyfile you write is actually just a convenience layer (syntactic sugar). Caddy’s core engine only understands configuration in JSON format. When you reload Caddy, the Caddyfile is adapted into a giant JSON document stored directly in Caddy’s RAM memory.

You can read this active configuration memory state anytime through the Admin API to make sure your Caddyfile is interpreted correctly:

# 1. Download the entire active JSON configuration and format it using jq
curl -s http://localhost:2019/config/ | jq .

# 2. Check the list of registered HTTP servers
curl -s http://localhost:2019/config/apps/http/servers/ | jq 'keys'

# 3. Track the active routes on the main server (srv0)
# Helps see the actual middleware evaluation order in Caddy's memory
curl -s http://localhost:2019/config/apps/http/servers/srv0/routes/ | \
    jq '[.[] | {id: .["@id"], match: .match[0].host, handlers: [.handle[].handler]}]'

# 4. Compare the active configuration with the local file to detect deviations
caddy adapt --config /etc/caddy/Caddyfile | jq . > /tmp/expected.json
curl -s http://localhost:2019/config/ | jq . > /tmp/actual.json
diff --color /tmp/expected.json /tmp/actual.json

Debugging Routing Matchers #

Routing problems are among the most frequent: you write a named matcher, but client requests don’t enter the handle block you want and instead fall into the default fallback.

The best debugging tactic is injecting custom response headers (header X-Debug-Route) dynamically in every handler block. This lets you visually see which route Caddy executed just by checking the response headers from the client:

example.com {
    # Define the matchers
    @api_route path /api/*
    @static_route path /static/* /assets/*
    @admin_route host admin.example.com
    
    # 1. Test the API Route
    handle @api_route {
        # Inject the debug header before proxying
        header X-Debug-Route "api-handler"
        reverse_proxy api:8080
    }
    
    # 2. Test the Static File Route
    handle @static_route {
        header X-Debug-Route "static-handler"
        file_server {
            root /var/www/static
        }
    }
    
    # 3. Test the Admin Subdomain Route
    handle @admin_route {
        header X-Debug-Route "admin-handler"
        reverse_proxy admin:9000
    }
    
    # 4. Default Fallback
    handle {
        header X-Debug-Route "fallback-handler"
        respond "No route matched by Caddy Matcher" 404
    }
}

After the configuration is enabled, use curl -I to verify the routing paths:

# Test the API path
curl -I https://example.com/api/users
# The output should show: X-Debug-Route: api-handler

# Test the static path
curl -I https://example.com/static/logo.png
# The output should show: X-Debug-Route: static-handler

# Test the fallback
curl -I https://example.com/unknown-path
# The output should show: X-Debug-Route: fallback-handler

Debugging Placeholder Values #

Caddy placeholders (written with curly braces {}) are how Caddy presents dynamic variables (like remote IPs, request headers, or JWT claims). If the placeholder is empty or mistyped, the variable won’t be exposed.

You can debug placeholder value contents by injecting them as temporary response headers:

example.com {
    # Expose placeholder values to HTTP response headers for debugging convenience
    header {
        X-Debug-Client-IP    "{remote_host}"
        X-Debug-Host         "{host}"
        X-Debug-URI          "{uri}"
        X-Debug-Method       "{method}"
        X-Debug-Scheme       "{scheme}"
        X-Debug-Request-UUID "{http.request.uuid}"
    }
    
    reverse_proxy localhost:3000
}

# Test the request and display all our custom headers
curl -I https://example.com/api/test?user=john | grep X-Debug

# Example output:
# X-Debug-Client-IP: 203.0.113.10
# X-Debug-Host: example.com
# X-Debug-URI: /api/test?user=john
# X-Debug-Method: GET
# X-Debug-Scheme: https
# X-Debug-Request-UUID: f3b0c4d8-c9ea-4b2a-8d1e-92718c3a9f0e

Debugging TLS and SSL Handshakes #

If the SSL/TLS handshake fails, client browsers can’t make a secure connection to the server. You must isolate whether the problem is at the certificate level (expired/untrusted), cipher protocol support, or SNI mismatch.

# 1. Display the complete SSL certificate information served by Caddy
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | \
    openssl x509 -noout -text | grep -E "Subject:|DNS:|Not After"

# 2. Check the list of supported cipher suites and TLS protocols
# Very important if you have legacy clients needing TLS 1.2
openssl s_client -connect example.com:443 -brief

# 3. Test whether Caddy responds correctly on the TLS 1.3 protocol
curl -v --tlsv1.3 https://example.com 2>&1 | grep "SSL connection"

# 4. Check Caddy's local Root CA validity period (if using local self-signed)
curl -s http://localhost:2019/pki/ca/local | jq '{name: .name, root_expires: .root.not_after}'

Debugging Reverse Proxy Latency #

When an application feels slow, clients often blame the web server. You must precisely measure whether the delay happens at the Caddy gateway transmission level, or because your backend is slow to produce data (TTFB delay).

# 1. Monitor backend (upstream) health status in real time via the Admin API
curl -s http://localhost:2019/reverse_proxy/upstreams/ | \
    jq -r '.[] | "\(.address): \(if .healthy then "UP" else "DOWN" end) (active connections: \(.num_requests), fails: \(.fails))"'

# 2. Compare the request execution time directly to the backend vs through Caddy
# Test directly to the backend (e.g., port 3000)
time curl -o /dev/null -s http://localhost:3000/api/heavy-job

# Test through Caddy (port 443 with SSL)
time curl -o /dev/null -s https://example.com/api/heavy-job

# The total time comparison (time_total) shows Caddy's overhead.
# In normal environments, Caddy's overhead is just a millisecond fraction (< 1ms).

Debugging Header Manipulation Issues #

Sometimes you think you’ve modified request headers using the header_up or header_down directives, but the backend reports not receiving those headers.

To verify this apple-to-apple, you can temporarily route traffic to a diagnostic echo server (like httpbin.org or the containous/whoami docker container) that returns all received request header data as the response body:

# Temporary test configuration
debug-header.example.com {
    reverse_proxy https://httpbin.org {
        # Change the host header so httpbin recognizes the request
        header_up Host {upstream_host}
        
        # Our custom header we want to debug
        header_up X-My-Custom-Header "Production-Token-Value"
        header_up X-Request-ID {http.request.uuid}
    }
}

# Send a request and see whether our custom header appears in httpbin's returned JSON
curl -s https://debug-header.example.com/headers | jq .
# In the JSON output, check the "headers" block to make sure X-My-Custom-Header exists.

Debugging with Caddyfile Adaptation (caddy adapt) #

The Caddyfile is a human-friendly configuration format. Before applying it to a production server, you can check how Caddy compiles that file into the native JSON format. This is very useful for detecting hidden routes or unexpected behavior caused by directive ordering.

# Adapt the local configuration file and display the formatted JSON
caddy adapt --config /etc/caddy/Caddyfile | jq .

# Track a certain route (e.g., the route for the example.com domain)
caddy adapt --config /etc/caddy/Caddyfile | \
    jq '.apps.http.servers.srv0.routes[] | select(.match[0].host[]? == "example.com")'

Creating a Minimal Reproducible Config #

If you find a bug or strange behavior that’s hard to isolate in a complicated production Caddyfile, create a minimal reproducible configuration standalone file. This strips all external configuration noise and focuses only on the one problematic feature.

# 1. Create a minimal test Caddyfile in the temporary folder
cat << 'EOF' > /tmp/test-caddy.Caddyfile
{
    # Run the admin API on a non-standard port to avoid clashes
    admin localhost:2020
    # Enable debug logging
    debug
}

# Use a plain localhost port without automatic SSL for easier local testing
localhost:8080 {
    # Use a dynamic mock response
    respond "Debug Server OK — Client IP: {remote_host}" 200
}
EOF

# 2. Run an isolated Caddy instance using the file above
caddy run --config /tmp/test-caddy.Caddyfile

# 3. In a separate terminal, test the response
curl http://localhost:8080/

After the minimal route works correctly, add middleware lines one by one from your production Caddyfile until the strange behavior reappears. This way immediately shows which line is the problem source.

Using the respond Directive for Mocking #

When troubleshooting routing, you’re often disturbed by dead or slow backends. You can use the respond directive as a temporary mock handler to make sure your Caddyfile named matchers filter requests correctly:

example.com {
    # ANTI-PATTERN: Writing reverse_proxy directly while debugging routing
    # reverse_proxy localhost:3000
    
    # CORRECT: Use temporary respond to verify the matcher
    @api_path path /api/v1/*
    handle @api_path {
        respond "ROUTE_MATCH: API Version 1" 200
    }
    
    @web_path path /web/*
    handle @web_path {
        respond "ROUTE_MATCH: Web App Frontend" 200
    }
    
    handle {
        respond "ROUTE_MATCH: Default Fallback" 404
    }
}

Test the responses with curl:

curl https://example.com/api/v1/users  # Output: ROUTE_MATCH: API Version 1
curl https://example.com/web/dashboard  # Output: ROUTE_MATCH: Web App Frontend

After the routing is proven to work accurately, replace the respond blocks back with your reverse_proxy or file_server directives.

Systematic Debug Diagnostic Script #

When handling a problematic production server, you need OS and Caddy information quickly in one command. The script below compiles all important diagnostic data into one text report:

#!/bin/bash
# caddy-diagnostic.sh — Caddy diagnostic data collection script
set -u

REPORT_FILE="/tmp/caddy-diagnostic-report.txt"

echo "=== STARTING CADDY DIAGNOSTICS ===" | tee "$REPORT_FILE"
echo "Test Time: $(date)" | tee -a "$REPORT_FILE"
echo "-----------------------------------" | tee -a "$REPORT_FILE"

echo "" >> "$REPORT_FILE"
echo "--- 1. CADDY VERSION ---" >> "$REPORT_FILE"
caddy version >> "$REPORT_FILE" 2>&1

echo "" >> "$REPORT_FILE"
echo "--- 2. SYSTEMD SERVICE STATUS ---" >> "$REPORT_FILE"
systemctl is-active caddy >> "$REPORT_FILE" 2>&1
systemctl status caddy --no-pager -n 10 >> "$REPORT_FILE" 2>&1

echo "" >> "$REPORT_FILE"
echo "--- 3. PORT NET STATS ---" >> "$REPORT_FILE"
sudo ss -tlnp | grep -E "caddy|2019|80|443" >> "$REPORT_FILE" 2>&1

echo "" >> "$REPORT_FILE"
echo "--- 4. REVERSE PROXY UPSTREAM ACTIVITY ---" >> "$REPORT_FILE"
curl -s http://localhost:2019/reverse_proxy/upstreams/ | jq . >> "$REPORT_FILE" 2>&1 || echo "Admin API unreachable" >> "$REPORT_FILE"

echo "" >> "$REPORT_FILE"
echo "--- 5. LAST 10 ERROR LINES IN JOURNALD ---" >> "$REPORT_FILE"
sudo journalctl -u caddy -n 50 --no-pager | grep -E "error|warn|fail" | tail -10 >> "$REPORT_FILE" 2>&1

echo "" >> "$REPORT_FILE"
echo "-----------------------------------" >> "$REPORT_FILE"
echo "[✓] Diagnostics complete. The report is saved at: $REPORT_FILE"

Run the script above when a problem occurs to get a comprehensive picture of your Caddy state in seconds.

Summary #

  • Verbose Debug — Enable the debug level globally in the Caddyfile or use the Admin API PUT command to raise log verbosity instantly without stopping traffic.
  • Admin API Inspection — Use the /config/ REST API to see Caddy’s current memory contents to detect discrepancies between the local Caddyfile and the active configuration.
  • Header Debugging — Inject custom response headers like X-Debug-Route or placeholder variables to verify route matching decisions directly from the client side.
  • httpbin Echo Testing — Test request header manipulation by temporarily pointing reverse_proxy to httpbin.org to see the actual headers Caddy sends.
  • Minimal Configuration — Isolate complicated configuration bugs by creating a minimal standalone Caddyfile in the /tmp/ directory and running it locally using caddy run.
  • respond Mocking — Use the respond directive as a temporary backend replacement to test named matcher filter logic without disturbance from upstream server failures.

← Previous: Common Errors   Next: Caddy Validate →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact