Error Log #

The error log is the internal recording system used by the Caddy web server to document operational problems, module failures, system errors, and abnormality warnings occurring during Caddy’s runtime. Fundamentally different from the access log, which records every successful or failed HTTP transaction interaction from external clients, the error log focuses exclusively on the health of Caddy’s own internal organs. If Caddy fails to update automatic SSL/TLS certificates from Let’s Encrypt, if the proxy connection to the backend application server suddenly drops, or if the Caddyfile configuration file fails to load during a restart, everything is recorded here. Having a deep understanding of the error log is essential for diagnosing network reliability problems, proactively monitoring certificate failures, and speeding up the troubleshooting process in production environments. We’ll discuss the structural differences between access logs and error logs, configure the global logging block, analyze real-world error message categories, enable deep debugging mode, and integrate the logging system with OS log management daemons.

Systematic Differences Between Access Logs and Error Logs #

Beginner system administrators often mix access log and error log data into a single unstructured file. This is a bad practice because it complicates log analysis using automated machines (parsing engines).

Here’s a table of fundamental characteristic differences between the two:

Global Logging Configuration and Verbosity Levels #

In Caddy, all internal logging configuration is managed at the top level inside the Global Options block at the very top of the Caddyfile. This is done because the system logging module initialization process must run earlier before Caddy parses your website domains.

The basic syntax of global logging configuration in the Caddyfile global options is:

{
    log {
        output <writer>
        format <encoder>
        level  <severity_level>
    }
}

Understanding Log Severity Levels #

Caddy applies standard log levels classification to filter messages based on importance. You can set the level parameter to control how much information you want written to the log file:

1. DEBUG: The most verbose level. Records every detailed internal binary activity (like TLS negotiation, route header matches, or buffer allocations). Only used in development environments or when solving complicated problems because it writes very large data.
2. INFO (Default): Records standard operational messages (like successful config reloads, server initialization starts, or new TLS certificates successfully issued). Very suitable for normal production operations.
3. WARN (Warning): Records abnormal conditions that don’t stop Caddy from running, but need attention (e.g., a connection to the backend briefly failed but succeeded on retry, or slow SSL responses).
4. ERROR (Error): Records critical function failures directly impacting client services (e.g., the backend is completely dead so a 502 Bad Gateway status is returned, or SSL certificates can’t be renewed).
5. PANIC / FATAL: Emergency levels indicating the Caddy server experiences fatal kernel-level or runtime failures causing the application to completely die.

Let’s learn the log level filtering flow in the following flowchart:

flowchart TD
    A["1. Internal Event Occurs in Caddy\n(e.g., dial tcp backend refused)"] --> B["2. Determine the Event Severity Level\n(e.g., Level = ERROR)"]
    B --> C{"3. Check the Global 'level' Configuration in the Caddyfile\n(e.g. Set level = WARN)"}
    
    C --> D{"Is the Event Level >= the Configuration Level?\n(In this case, ERROR >= WARN?)"}
    
    D -- "Yes" --> E["4. Compose a structured error log message"]
    D -- "No" --> F["5. Ignore the event (Don't write to the log)"]
    
    E --> G["6. Send to the Error Log Output Module\n(e.g., stderr / systemd journald)"]

Error Log Configuration Example in the Global Options #

Here’s an industry-standard Caddyfile configuration for separating Caddy error logs independently with automatic file rotation:

# 1. Caddyfile Global Options Block
{
    log {
        # Write all Caddy error logs to a separate file
        output file /var/log/caddy/caddy_system.log {
            roll_size 50mb
            roll_keep 10
        }
        # Use structured JSON format for monitoring compatibility
        format json
        
        # Only record WARN level events and above in this file
        level warn
    }
}

# 2. Normal Site Route Block
example.com {
    reverse_proxy localhost:8080
}

Module-Name-Specific Logging Tuning (Logger Filtering) #

Caddy divides its internal log structure into various logger categories based on the module triggering that activity (e.g., http.handlers.reverse_proxy, tls, admin.api, or http.acme_client).

If you set the global severity level to ERROR to save disk capacity, but you’re troubleshooting a backend proxy integration problem and need DEBUG level verbosity only for the reverse proxy module, Caddy lets you set specific log level rules per logger name:

# Example: Isolating Debug Verbosity Specifically for the Reverse Proxy Module
{
    # 1. General system error log (WARN level and above)
    log system_errors {
        output file /var/log/caddy/system.log
        level warn
    }

    # 2. Reverse proxy investigation log (Special DEBUG level)
    log proxy_debug {
        output file /var/log/caddy/proxy_debug.log
        level debug
        
        # Only certify the reverse proxy module logger to this file
        include http.handlers.reverse_proxy
    }
}

example.com {
    reverse_proxy localhost:8080
}

Analyzing Real-World Caddy Error Categories #

As a server administrator, you must recognize the typical error messages that often appear in Caddy error log files to speed up system recovery mitigation.

1. Upstream Connection Failures (Reverse Proxy Connection Failure) #

This error message occurs when Caddy fails to reach the backend application server (e.g., Node.js or PHP-FPM) you defined in the reverse_proxy directive.

Example JSON Error Log Text:

{
  "level": "error",
  "ts": 1781609500.654,
  "logger": "http.handlers.reverse_proxy",
  "msg": "aborting request due to backend write failure",
  "error": "dial tcp 127.0.0.1:8080: connect: connection refused"
}

• Problem Meaning: The backend application on port 8080 is dead, crashed (out of memory), or isn’t running on that local loopback IP address.
• Remedial Action: Check your backend application runtime status using systemctl status or rerun the application container process.

2. TLS / ACME Challenge Issues (TLS Certificate Failure) #

Occurs when Caddy experiences failures trying to obtain or renew free SSL/TLS certificates from Let’s Encrypt or ZeroSSL.

Example JSON Error Log Text:

{
  "level": "error",
  "ts": 1781609510.987,
  "logger": "http.acme_client",
  "msg": "challenge failed",
  "challenge_type": "http-01",
  "error": "accepting challenge: HTTP 400 Bad Request - urn:ietf:params:acme:error:connection - Connection refused"
}

• Problem Meaning: The Let’s Encrypt challenge server tries accessing port 80 (HTTP) on your Caddy server to validate domain ownership, but the connection is refused.
• Remedial Action: Make sure port 80 on your Caddy server is openly accessible at the network firewall level (like AWS Security Groups or cloud provider firewalls) and isn’t blocked by your ISP.

3. SNI Mismatch Issues (Unknown TLS Client Hello) #

This error message is often triggered when IP scanner network bots try contacting your Caddy server directly via IP address without sending a valid domain name (SNI).

Example JSON Error Log Text:

{
  "level": "warn",
  "ts": 1781609520.123,
  "logger": "tls.handshake",
  "msg": "no certificate available for the requested server name",
  "server_name": "203.0.113.10"
}

• Problem Meaning: Caddy’s TLS server rejects the encryption handshake because it doesn’t have a valid SSL certificate for the domain named by the IP address 203.0.113.10.
• Remedial Action: This is a normal production warning caused by internet bot activity. It doesn’t require mitigation unless the bot request volume is very large (DDoS), which can be dampened using IP blocklists.

Handling Panics and Go Runtime Stack Traces #

Caddy is written in the Go programming language (Golang). One characteristic of Go applications is when a very severe low-level problem occurs inside the memory code body — for example running out of binary RAM (out of memory panic) or simultaneous map write failures by goroutines (concurrent map writes) — the Go runtime experiences a panic and prints the entire function call stack (stack trace).

// Illustration of the Go Runtime Panic output format in the Log
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x10b2df4]

goroutine 127 [running]:
github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP(0xc00045f800, 0x140e700, ...)
    /go/pkg/mod/github.com/caddyserver/caddy/[email protected]/modules/caddyhttp/reverseproxy/reverseproxy.go:342 +0x234

• Problem Meaning: This isn’t just a common Caddyfile configuration error. This is a fatal software bug or system crash causing the Caddy binary process to die instantly.
• Rescue Action: Caddy can’t recover itself from a panic condition. You must use an external supervisor system (like the systemd daemon on Linux with Restart=on-failure configuration or the Docker restart policy unless-stopped) to ensure your Caddy server is automatically restarted by the OS within seconds after a crash.

Enabling Deep Debugging Mode (Verbose Logging) #

During local development or when your Caddy server routes aren’t responding as expected (e.g., requests being routed to the wrong server block), you must enable level debug to see Caddy’s internal calculation processes.

# Caddyfile Example for Local Debugging
{
    log {
        output stderr
        format console
        # Enable very verbose debug level
        level debug
    }
}

example.com {
    @restricted {
        header X-Special-Token "secret"
    }
    
    # Special route whose behavior we want to investigate
    handle @restricted {
        respond "Special Access Accepted" 200
    }
    
    handle {
        respond "Public Access" 200
    }
}

By enabling level debug and directing the output to stderr (displayed directly on the terminal console), Caddy records every named matcher evaluation step:

• Caddy writes: @restricted matcher evaluated: Header match false (Value did not match)
• This greatly helps you detect whether route errors are caused by header name typos, hidden spaces, or parameter type mismatches.

Real-Time Error Alerts and Notification Integration #

Writing error logs to local files is a good first step, but it’s useless if no one reads them until your website dies. In critical production environments, you must integrate Caddy error logs with automated alert systems.

You can use the Promtail utility (from the Grafana ecosystem) to continuously scan the /var/log/caddy/system.log file. If it detects JSON entries with the "level": "error" field, Promtail sends metrics to Prometheus, which then triggers Alertmanager to instantly send notifications to your developer team’s Slack channel:

# Example Promtail Alerting Pipeline
pipeline_stages:
  - json:
      expressions:
        level_value: level
        logger_name: logger
        error_msg: error
  - match:
      selector: '{job="caddy-system"} | json | level_value = "error"'
      stages:
        - metrics:
            caddy_error_counter:
              type: Counter
              description: "Total number of critical Caddy system errors"
              source: level_value
              config:
                action: inc

This way, your operations team can immediately identify server anomalies (like upstream DNS resolution failures) within seconds before triggering complaints from your application’s end users.

Defensive Actions Against Bloating Error Logs #

On high-traffic production servers, error log files including the WARN level can bloat from internet bot request streaks triggering TLS warnings (like SNI mismatches). You must know how to defensively limit this logging so it doesn’t consume server disk capacity:

1. Use the ERROR Log Level in Production: For public servers directly exposed to the internet without a Cloudflare firewall in front, avoid setting INFO or WARN levels as the global default. Use level error so Caddy only records functionality failure events truly critical to your backend applications.
2. Automatic Log File Rotation: Always configure log rotation parameters (roll parameters) on your file output blocks, e.g., limiting the maximum size per file to 50mb and keeping a maximum of 5 old backups to secure disk I/O.
3. Network Edge Firewall Optimization: Dampen network scanner bot traffic at the outermost level (like AWS Security Groups or Cloudflare IP Firewalls) before it reaches your Caddy server’s TCP socket system, which automatically cuts the TLS error log records polluting the system.

Summary #

• Main Function: Error logs record Caddy’s internal operational status like TLS failures, Caddyfile loading errors, and reverse proxy dial failures.
• Global Configuration: Error log configuration is managed centrally at the top-level Global Options block of the Caddyfile.
• Severity Levels: Caddy classifies log severity from lowest to critical: DEBUG, INFO, WARN, ERROR, PANIC, and FATAL.
• Logger Isolation: You can isolate special debug verbosity for specific modules (like http.handlers.reverse_proxy) to prevent log bloating.
• Debugging Mode: Enable level debug to investigate named matcher evaluations and TLS handshake failures in local development environments.
• TLS & ACME Alerts: Proactively monitor error logs to detect Let’s Encrypt ACME challenge failures before certificates expire.
• Defensive Actions: Limit log size in production by setting level error to filter out TLS warning log streaks caused by internet bot activity.
• systemd Supervisor: Go Runtime Panics require an external systemd supervisor with the Restart=on-failure option so Caddy automatically restarts after a binary crash.

← Previous: Access Log   Next: Format Log →