Format Log #

Log format is the visual representation and data structure of log records produced by the Caddy server before being sent to the log output target. Correct log format configuration acts as an important bridge between Caddy’s raw binary system data and your monitoring systems’’ intelligence. By default, Caddy provides a structured JSON format ideal for data-processing machines, plus a custom console format friendly to human eyes during local development. However, at production scale, you’re often required to modify that log format: discarding unused data fields to save bandwidth, renaming fields to match your central database schema, hiding sensitive query parameters, and masking client IP addresses to comply with global data privacy regulations like GDPR. We’ll thoroughly dissect Caddy’s log encoder architecture, compare JSON and console formats, learn sensitive data filtering and anonymization techniques, practice using field selectors, and compose log format transformations for high compatibility.

Caddy’s Log Encoder Architecture #

In Caddy’s internal server system, log recording isn’t done monolithically or directly written to storage media files. Caddy divides the logging process into two separate, loosely collaborating modules: Writer and Encoder. The Writer has the single responsibility of determining where the log data stream is sent (e.g., to a local disk file, the standard output console, or streamed over the network to a syslog server). Meanwhile, the Encoder holds the exclusive responsibility of defining how the structured event data is converted into a text or binary representation before being handed to the Writer.

When an HTTP request arrives or a system failure occurs, Caddy’s logging subsystem (built on Go’s high-speed structured logging library named zap) collects all event metadata in the form of a structured data structure in RAM. This data structure contains various information from high-precision timestamps, severity levels, logger names, main messages, to detailed HTTP request and response data. After the data is collected, Caddy calls the Encoder module configured in the Caddyfile. This Encoder then traverses all that binary data, applies the manipulation filters you requested, and serializes it into a formatted string (e.g., a compressed JSON line or console text with level-marker colors). This serialized text string result is what’s then handed to the Writer to be stored to disk or sent to the network.

This architectural separation between formatting (Encoder) and I/O writing (Writer) provides incredible flexibility. You can change the log format from JSON to plain text, hide sensitive user data, or modify time formats without worrying about how those log files are rotated, compressed, or sent outside the server.

Integrated Log Formats: JSON vs Console vs Logfmt #

Caddy provides three built-in log encoder modules ready to use without installing external modules. Each encoder has unique characteristics designed for different deployment scenarios.

1. JSON Encoder (format json) #

The JSON encoder is the standard choice and most recommended for all production environments. This format writes every log entry as a single-line compressed JSON object. The JSON encoder preserves the native data types of every metric — like integer numbers for response statuses, decimal float numbers for processing durations, string arrays for HTTP headers, and booleans for TLS security statuses.

• Why Choose JSON: Modern log aggregator engines like Elasticsearch, Grafana Loki, Datadog, Splunk, and AWS CloudWatch can directly read and index JSON objects efficiently without requiring complicated regular expression (regex) parsing configurations. This speeds up log searching and real-time monitoring metric dashboard creation.
• JSON Weaknesses: Very hard to read directly by human eyes on terminal screens because text lines have no indentation spacing (minified) to save storage space.

Here’s an example of the raw access log line produced by Caddy’s JSON encoder:

{"level":"info","ts":1781609400.1234,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"203.0.113.50","proto":"HTTP/2.0","method":"GET","host":"example.com","uri":"/api/users","headers":{"User-Agent":["curl/7.81.0"],"Authorization":["Bearer secret_token"]}},"duration":0.0045,"size":142,"status":200}

2. Console Encoder (format console) #

The console encoder is specifically designed to increase your productivity during local development processes. This format translates the structured in-memory objects into neat columnar text lines, formats decimal Unix timestamps into easy-to-read local time, and provides terminal ANSI color coding effects on log severity levels (yellow for warnings, red for errors, green for info).

• Why Choose Console: You can quickly read incoming request flows, detect HTTP errors, and read debug messages directly on your interactive console terminal without additional parsing tools.
• Console Weaknesses: Very inefficient for production environments. Creating colored console text consumes higher CPU cycles, discards some detailed metadata for readability aesthetics, and makes it very hard for log shipper agents to do automatic query analysis.

Here’s the visual representation of the same log entry when formatted using the Console encoder:

2026-06-16T18:44:15.123+0700    INFO    http.log.access    handled request    {"request": {"remote_ip": "203.0.113.50", "proto": "HTTP/2.0", "method": "GET", "host": "example.com", "uri": "/api/users"}, "duration": 0.0045, "status": 200}

3. Logfmt Encoder (format logfmt) #

The logfmt encoder formats log data into a series of space-separated key-value pairs. Values containing spaces or special characters are automatically wrapped in double quotes.

• Why Choose Logfmt: Very popular in the Go, Kubernetes, and Heroku ecosystems. This format offers a good middle ground: easier for human eyes to read than raw JSON, yet still very easy to parse efficiently by log collection engines like Grafana Loki or Fluentd.
• Logfmt Weaknesses: Nested data structures like Caddy’s HTTP request objects must be flattened into a one-dimensional format (e.g., request.method=GET), which can sometimes be confusing if the data structure is very complex.

Here’s an example of the same log line formatted in Logfmt:

ts=1781609400.1234 level=info logger=http.log.access msg="handled request" method=GET host=example.com uri=/api/users duration=0.0045 status=200

Comparing Code: When to Choose the Right Format? #

Let’s study the Caddyfile comparison example below to see common configuration mistakes (anti-patterns) and their solutions:

# Log format writing comparison example based on environment
example.com {
    # ANTI-PATTERN: Using the console format in large-scale production environments.
    # This slows down disk I/O writing because of colorful text formatting,
    # and makes it hard for log aggregators (Loki/Elasticsearch) to parse the data.
    log {
        output file /var/log/caddy/production.log
        format console
    }
    
    # CORRECT: Use structured JSON format for production environments.
    # This ensures efficient log data retention, minimal CPU usage,
    # and 100% compatibility with log aggregators.
    log {
        output file /var/log/caddy/production.json.log
        format json
    }
}

JSON Format Customization: Timestamps #

By default, Caddy’s JSON encoder records event time (timestamps) in decimal Unix epoch fraction number format (e.g., "ts": 1781609400.1234). This format records the number of seconds elapsed since January 1, 1970 UTC. For computers and databases, this format is ideal because number sorting operations are much faster than date string sorting. However, for us humans, this format is very confusing when manually reading logs for quick investigations.

To overcome this, Caddy lets you customize the timestamp representation inside the JSON format. There are several time formats supported by Caddy:

Here’s a Caddyfile implementation example for changing the time format to the international standard ISO-8601, highly recommended for monitoring integration:

# Caddy JSON time format customization example
example.com {
    log {
        output file /var/log/caddy/access.log
        format json {
            # Customize the time format to the international ISO8601 standard.
            # Highly recommended because Elasticsearch and Loki directly recognize it
            # as the main index timestamp without extra processing.
            time_format "iso8601"
            
            # We can also use the "wall" format if we want to monitor logs
            # on the local server with the local server timezone.
            # time_format "wall"
        }
    }
    reverse_proxy localhost:8080
}

Log Field Manipulation: Deleting and Obscuring Sensitive Data #

In real modern web applications in production environments, user data security and confidentiality are the number one priority. By default, Caddy’s access logs record HTTP request header information in full. A serious problem arises when those requests carry sensitive data like authentication tokens in the Authorization header, authentication cookies in the Cookie header, or user personal data in other header parameters. If this data is written directly to plain text log files on the server, your server violates global privacy regulation compliance like GDPR (European Union) or PCI-DSS (credit card security standards).

To prevent this sensitive data leakage, Caddy provides robust log field manipulation features through a direct filter mechanism inside the Caddyfile encoder configuration.

1. Deleting Sensitive Fields (filter delete) #

You can delete certain fields from the JSON log structure so they’re never converted to strings and never touch your server’s hard disk. You do this using the filter keyword followed by a field selector and the delete action.

# Example of sensitive header deletion for security compliance
example.com {
    log {
        output file /var/log/caddy/access.log
        format json {
            # Delete authentication tokens from access request logs
            filter request>headers>Authorization delete
            
            # Delete Cookie data from request logs to prevent session leakage
            filter request>headers>Cookie delete
            
            # Delete response Cookie data from our backend web
            filter resp_headers>Set-Cookie delete
        }
    }
    reverse_proxy localhost:8080
}

In the example above, the greater-than character (>) acts as a nested path selector to locate the header data position inside Caddy’s JSON log object.

[!CAUTION] DON’T use the dot (.) character as a field selector separator. Developers accustomed to JSONpath syntax (like request.headers.Cookie) often wrongly write selectors in the Caddyfile with dot characters. The Caddyfile must use the greater-than character (>) as the token separator. If you write a dot (e.g., filter request.headers.Cookie delete), the Caddyfile parser treats request.headers.Cookie as a single flat field name, so the filter fails to work and your sensitive data still leaks to disk.

2. Anonymizing Visitor IP Addresses (filter ip_mask) #

Under the EU GDPR data protection regulation, visitor IP addresses are considered Personally Identifiable Information (PII) because they can theoretically be used to track individual identities. Storing visitors’ full IP addresses without encryption or strict consent is a legal violation. However, you still need IP address information for security statistics analysis (like detecting DDoS attacks or knowing visitor country origins).

The best solution is anonymizing IP addresses by masking part of the IP address bits before writing to the log file. Caddy provides a very efficient built-in ip_mask filter for this purpose:

# IP address anonymization configuration example for GDPR compliance
example.com {
    log {
        output file /var/log/caddy/access.log
        format json {
            # Mask the client IP address with a CIDR mask:
            # - For IPv4: use a 16-bit mask (hides the last 2 octets, e.g. 203.0.113.50 becomes 203.0.0.0)
            # - For IPv6: use a 32-bit mask (hides most of the hexadecimal blocks)
            filter request>client_ip ip_mask 16 32
            filter request>remote_ip ip_mask 16 32
        }
    }
    reverse_proxy localhost:8080
}

With the configuration above:

• The IP address 203.0.113.50 is stored in the log file as 203.0.0.0.
• You can still know the visitor came from that regional internet provider block without violating their personal data privacy because the original IP data is permanently removed in RAM before it can be written to disk storage.

Log Enrichment: Injecting Custom Data into Logs #

You often need to track an HTTP request’s journey across your entire system architecture (distributed tracing). To ease this tracking, you must inject a Unique Request ID into Caddy’s access logs and the request header sent to your backend application servers.

Caddy lets you leverage dynamic variables (placeholders) to inject custom data into request headers. Because Caddy logs record request header contents automatically, this header addition action directly enriches the information inside your access logs:

# Log Enrichment configuration example with a custom Request ID
example.com {
    log {
        output file /var/log/caddy/access.log
        format json
    }

    # Take the unique UUID automatically created by Caddy for each request,
    # then put it into the X-Request-ID request header.
    # This header is automatically recorded under request>headers>X-Request-Id in the JSON log.
    header_up X-Request-ID {http.request.uuid}
    
    # Insert the username info if the user successfully logged in using Basic Auth
    header_up X-Authenticated-User {http.auth.user.id}

    reverse_proxy localhost:8080
}

By applying the tactic above, your operations team can easily copy the X-Request-ID value from Caddy’s access logs and search it in your backend application’s log database to see the entire database execution trail triggered by that request.

Filtering Query Parameters Using Regex Filters #

Sensitive data leakage doesn’t only happen in HTTP request headers. Another classic problem is leaking secret tokens or API keys sent by client applications through URL query string parameters (e.g., requests to /api/auth/callback?token=secret12345).

By default, Caddy records these query string parameters inside the request>uri field. You can’t simply delete the request>uri field because you still need URL path information for application route analysis. The solution is filtering and replacing sensitive query value parts using a regular expression (regexp) filter:

# Sensitive token filtering example on log URL queries
example.com {
    log {
        output file /var/log/caddy/access.log
        format json {
            # Regexp filter to detect token parameters in URL queries
            # Syntax: filter <field> regexp <pattern> <replacement>
            # This detects token=[characters] and replaces it with token=REDACTED
            filter request>uri regexp "token=[a-zA-Z0-9]+" "token=REDACTED"
            
            # Additional filter to hide API keys
            filter request>uri regexp "api_key=[a-zA-Z0-9]+" "api_key=REDACTED"
        }
    }
    reverse_proxy localhost:8080
}

When the filter above is executed, the log entry for the request /users?token=secretXYZ&status=active is written as /users?token=REDACTED&status=active.

[!TIP] Watch Out for Regex Performance. Regular expression pattern matching requires more CPU computation power than plain string matching. If your Caddy server handles tens of thousands of requests per second, make the regex patterns as specific as possible and avoid greedy wildcards like .* that can trigger degraded Caddy server traffic handling performance.

LogQL Query Analysis on Grafana Loki #

After designing clean JSON log formats and sending them to a centralized log aggregator like Grafana Loki, you can use the power of the LogQL query language to do in-depth system monitoring without writing manual parsing scripts.

Here are some practical LogQL query examples for analyzing the structured JSON logs produced by Caddy:

1. Calculating Average Response Time (95th Percentile) #

You want to monitor your application’s response latency performance for API routes. LogQL can directly parse JSON objects and extract the duration field (in seconds) to be calculated as a metric:

# Calculate the 95th percentile request processing duration within a 5-minute interval
quantile_over_time(0.95, {job="caddy-access"} | json | unwrap duration [5m])

2. Detecting Server Error Spikes (HTTP 5xx) #

You want to monitor if a backend application suddenly dies and causes Caddy to return HTTP 502 Bad Gateway or 504 Gateway Timeout errors:

# Display all log entries with HTTP response status 500 and above
{job="caddy-access"} | json | status >= 500

3. Grouping Request Volume by Domain #

If your Caddy serves many domain names (multi-tenancy), you can create total requests-per-second charts grouped by host domain:

# Calculate total requests per second grouped by the request host field
sum by (request_host) (rate({job="caddy-access"} | json | __error__="" | unwrap request_host [1m]))

Log Formatting DSL: Single Field & Field Selectors #

Caddy provides very efficient tools for radically manipulating log formats without wasting CPU memory. This concept uses field selectors and a special DSL format.

1. Single Field Encoder (format single_field) #

Sometimes you don’t need large, complex structured JSON data. For example, when deploying Caddy inside minimalist containers (like Kubernetes pods) and you want Caddy to only output raw error message strings to stdout so an external supervisor system can directly process them:

# Single Field Encoder configuration example
example.com {
    log {
        output stdout
        # Only output the value of the "msg" field as plain text.
        # Output: "handled request" or "connection refused" without any JSON schema.
        format single_field msg
    }
    reverse_proxy localhost:8080
}

2. Caddy Field Selector Guide #

To make composing log manipulation filters easier, here’s a reference table for writing custom field selector paths fully supported by Caddy’s log parser:

Log Transformation Processing Flow Diagram by the Encoder #

To make understanding the log data processing cycle from raw data to a safe, clean string easier, let’s look at the processing pipeline diagram below:

flowchart TD
    A["1. Event Detected\n(Binary data in Caddy's RAM)"] --> B["2. Evaluate the Main Encoder Format\n(e.g., format json / console / logfmt)"]
    
    B --> C{"3. Is there a 'delete' filter?"}
    C -- "Yes" --> D["Cut the target field from memory\n(e.g. Delete the Authorization header)"]
    C -- "No" --> E{"4. Is there an 'ip_mask' filter?"}
    D --> E
    
    E -- "Yes" --> F["Mask the client IP address bits\n(e.g. Change 203.0.113.50 -> 203.0.0.0)"]
    E -- "No" --> G{"5. Is there a 'regexp' filter?"}
    F --> G
    
    G -- "Yes" --> H["Replace text values using regex\n(e.g. token=secret -> token=REDACTED)"]
    G -- "No" --> I["6. Convert the final object into a text string"]
    H --> I
    
    I --> J["7. Hand the formatted string to the Output module\n(Write to the access.log file via the Writer)"]

This transformation process guarantees log data modifications happen entirely inside the isolated RAM memory environment. The original sensitive data never gets written to your physical disk storage media, minimizing data leakage risks from physical server hacking attacks.

Summary #

• Role Separation: The Encoder handles log data visualization and structure in RAM, while the Writer handles the physical storage of that data to the final destination.
• Structured JSON: The best format for production environments, enabling 100% integration with Elasticsearch and Grafana Loki without extra regex.
• Interactive Console: A human-friendly format with level-marker colors, ideal to install only in your local development environment.
• Time Standardization: Use the time_format "iso8601" option inside the JSON encoder to standardize log search indexing in monitoring systems.
• GDPR Protection: Apply the ip_mask filter on request>client_ip to comply with global privacy regulations by masking visitor IP addresses.
• Credential Security: Always clean logs of sensitive authentication data by applying the delete filter on the Authorization and Cookie headers.
• URL Query Redaction: Leverage the regexp filter to replace dynamic session token values in URL query string parameters with the word REDACTED.
• Field Selectors: Use the > separator character (not the dot) to navigate nested JSON object structures in the Caddyfile.
• Single Field: Use the single_field encoder if you want to present just one raw text metric to the container stdout to save processing load.

← Previous: Error Log   Next: Log Output →