Passive Health Check #

Periodically monitoring backend (upstream) server health using active testing methods (active health check) is proven effective for filtering out dead servers before they receive user traffic. However, in dynamic, large-scale production environments, active monitoring alone isn’t enough. There are various types of failures that only occur when the system is handling real workloads from real users — like database connections failing mid-request, sudden memory leaks slowing down query processing, or deadlocks in application processing threads. This is where the Passive Health Check plays an important role. We’ll discuss how passive monitoring works in Caddy, the details of all failure detection configuration parameters (including HTTP statuses and latency), the built-in Circuit Breaker integration, synergy with retry logic for maximum fault tolerance, and a tactical comparison between active and passive approaches to ensure high reliability for your infrastructure.

How Passive Health Check Works #

The Passive Health Check mechanism works on the real traffic observation principle. Unlike active health checks that periodically send artificial test packets (probes), passive monitoring generates zero additional traffic on your network. Caddy acts as a passive observer monitoring every response returned by backend servers to users.

Each time a user sends an HTTP request, Caddy forwards it to one of the backend servers selected by the load balancer policy. During this transmission and processing, Caddy monitors the TCP socket connection status, SSL/TLS handshake, backend response speed, and the returned HTTP status code.

Detection and Recovery Cycle #

When a backend failure occurs, Caddy records the incident internally in memory. This passive detection workflow follows this cycle:

1. Real Performance Monitoring: Caddy forwards requests from site visitors to backend servers.
2. Failure Tracking: If the backend fails to respond (e.g., unilateral TCP connection termination or returning a 503 error code), Caddy increments the failure counter for that backend.
3. Threshold Activation: If the failure count exceeds the safe limit you set (max_fails) within a certain time window (fail_duration), Caddy immediately marks that backend as unhealthy.
4. Backend Isolation: Caddy removes the problematic backend from the active load balancer rotation list. For the specified fail_duration, Caddy won’t send new user requests to that server.
5. Trial Recovery: After the fail_duration isolation time expires, Caddy passively gives that backend a chance to prove itself again. When the next request arrives, Caddy tries routing it to that backend. If the response succeeds, the failure count is cleared and the backend is declared healthy again. However, if the response still fails, the backend is immediately re-isolated for another fail_duration.

Passive Monitoring Workflow Representation #

For a clearer picture, let’s look at the flow diagram of how Caddy handles user requests and passively records failures on the upstream cluster:

flowchart TD
    User["1. Visitor Sends Request"] --> Caddy{"2. Caddy Gateway"}
    Caddy -->|"Select Upstream"| Backend1["Backend Server 1 (Status: Healthy)"]
    Caddy -->|"Select Upstream"| Backend2["Backend Server 2 (Status: Unhealthy)"]

    subgraph Backend_Evaluation["Real-Time Passive Evaluation"]
        Backend1 -->|"Return 200 OK"| CheckSuccess{"Was it Successful?"}
        CheckSuccess -- "Yes" --> ReturnUser1["Send Reply to Visitor"]
        CheckSuccess -- "No (Failed)" --> IncrementFail1["Increment Fail Counter"]
        
        Backend2 -.->|"Receives Post-Isolation Trial Request"| CheckRecover{"Was it Successful?"}
        CheckRecover -- "Yes" --> ResetFail["Clear Fail Counter & Status: Healthy"]
        CheckRecover -- "No" --> ReIsolate["Re-isolate for the fail_duration"]
    end

    style Caddy stroke:#0288d1,stroke-width:2px
    style Backend1 stroke:#43a047,stroke-width:2px
    style Backend2 stroke:#e53935,stroke-dasharray:5,5

Passive Health Check Configuration Parameters #

In the Caddyfile configuration, passive monitoring is configured directly inside the reverse_proxy block. These parameters let you adjust the detection sensitivity and backend recovery duration:

# Basic passive health check configuration example
example.com {
    reverse_proxy backend-1:3000 backend-2:3000 {
        # Enable passive monitoring with a 30-second isolation duration
        fail_duration 30s
        
        # Maximum failure tolerance before isolation
        max_fails 3
        
        # Additional HTTP status code criteria considered as failures
        unhealthy_status 500 502 503 504
        
        # Additional response latency criteria considered as failures
        unhealthy_latency 5s
        
        # Maximum concurrent requests per upstream
        unhealthy_request_count 50
    }
}

Here’s an in-depth explanation of each subdirective used above:

1. fail_duration #

The fail_duration subdirective is the most important parameter in Caddy’s passive monitoring configuration. By default, fail_duration is 0 (zero), meaning the Passive Health Check feature is completely disabled.

When you set fail_duration to a valid duration value (e.g., 10s, 30s, or 5m), you instruct Caddy to enable failure tracking and remember each backend’s failure status for that duration. This parameter has a dual role:

• Tracking Time Window (Sliding Window): Caddy monitors the failure count occurring within this time range. If you set fail_duration 30s, Caddy only counts failures accumulated in the last 30 seconds. Failures older than 30 seconds expire and are removed from the count.
• Isolation Time (Penalty Duration): When a backend exceeds the maximum failure limit (max_fails), that backend is isolated from the load balancer rotation for this duration.

2. max_fails #

The max_fails subdirective sets the maximum number of failures allowed on one backend within the fail_duration window before it’s declared unhealthy. The default value for this parameter is 1.

• Sensitive Design (max_fails 1): Suitable for critical systems with high backend redundancy. A single failure on a backend server immediately triggers Caddy to isolate it from the network, minimizing the number of users affected by the failure.
• Tolerant Design (max_fails 3 or more): Useful when the network connection between Caddy and backend servers frequently experiences small temporary disruptions (network flapping). By setting a value greater than one, you avoid hastily isolating backends due to momentary network glitches.

3. unhealthy_status #

By default, Caddy considers a request failed only if a transport-level problem occurs (like a refused connection or socket timeout). However, your web applications often still return valid HTTP responses but with error status codes like 500 Internal Server Error or 503 Service Unavailable.

By adding unhealthy_status, you extend the failure detection scope to the application level. You can register specific HTTP status codes or use class formats (like 5xx) to detect application failures:

# ANTI-PATTERN: Ignoring application error status codes
reverse_proxy backend-1:3000 backend-2:3000 {
    fail_duration 30s
    # Caddy only detects OS/network crashes but keeps routing traffic to backends
    # experiencing internal database errors and returning status 500.
}

# CORRECT: Capturing internal server failures
reverse_proxy backend-1:3000 backend-2:3000 {
    fail_duration 30s
    max_fails 3
    unhealthy_status 500-504
}

[!WARNING] Never include 4xx status code classes (like 400 Bad Request or 404 Not Found) in the unhealthy_status configuration. 4xx codes represent client (end-user) errors, not backend server failures. If you include 4xx, users accidentally accessing nonexistent pages (404) could cause your healthy backend servers to be unfairly isolated.

4. unhealthy_latency #

When your backend servers are overloaded, one of the first symptoms is slower response times. The server doesn’t immediately die or return errors; it processes requests very slowly. This can cause connection queues to pile up on the Caddy side, ultimately degrading the entire application gateway’s performance.

The unhealthy_latency subdirective lets you detect those slowing backend servers. If a backend’s response takes longer than the specified duration value (e.g., unhealthy_latency 5s), Caddy counts that request as one failure.

5. unhealthy_request_count #

The unhealthy_request_count parameter sets the maximum number of concurrent connections (simultaneously processed requests) for each backend server. If a server is processing requests at or above this limit, Caddy considers the server saturated and temporarily marks it unhealthy so no additional load is sent to it. This greatly helps prevent backend collapse from the Thundering Herd phenomenon.

Event-Driven Failure Detection Mechanism #

To understand how Caddy intelligently identifies failures passively, we need to see how Caddy processes connections at the OS and network level. Caddy uses an event-driven failure handling model based on real events in the network protocol stack:

+-------------------------------------------------------------------------+
||                  Passive Failure Classification in Caddy               ||
+-------------------------------------------------------------------------+
||  1. Network & Transport Failures (Automatically Detected):            ||
||     * TCP Connection Refused (Backend dead / Port closed)             ||
||     * TCP Connection Timeout (Firewall blocking packets / Busy server)||
||     * TLS Handshake Timeout (Encryption negotiation stuck)            ||
||     * Read/Write Timeout (Connection dropped mid-data-transmission)    ||
||                                                                         ||
||  2. Application Failures (Optional via Configuration):                 ||
||     * HTTP Status Criteria (e.g., 500, 502, 503, 504)                 ||
||     * Response Latency Criteria (Response exceeds tolerance limit)    ||
+-------------------------------------------------------------------------+

Transport-Level Detection #

When Caddy tries to forward a user request to a backend server, it makes a system call to the OS kernel to open a new connection socket. Caddy observes the following transport failure types:

• TCP Connection Refused (RST Packet): If the backend application port is dead or the backend process isn’t running, the destination OS returns an RST (Reset) packet to Caddy. Caddy instantly detects this as a transport failure.
• TCP Connection Timeout: If the backend server is completely dead or there’s a network routing problem, the SYN packet Caddy sent never gets a reply. The connection hangs until the dial timeout ends, which Caddy immediately records as a failure.
• TLS Handshake Failures: If the backend is configured with HTTPS and a TLS certificate negotiation failure occurs — like an expired certificate, an untrusted one, or a handshake hanging past the timeout — Caddy considers the backend failed.

Application-Level Detection #

After the TCP and TLS connections are successfully established, Caddy sends the HTTP request payload and waits for the response. This is where the unhealthy_status and unhealthy_latency configurations are evaluated:

1. Status Code Evaluation: As soon as the HTTP response header is received from the backend, Caddy matches its status code against the unhealthy_status list. If there’s a match, Caddy increments the failure indicator.
2. Latency Timeout Evaluation: Caddy calculates the time from the first request byte sent to the first response byte received. If this duration exceeds the unhealthy_latency limit, Caddy stops the wait (or lets it finish depending on transport configuration) and records one failure.

The Integrated Circuit Breaker Pattern #

Circuit Breaker is a software architecture design pattern used to prevent local failures in one service from spreading to the entire system. Caddy elegantly integrates this Circuit Breaker pattern directly into its passive monitoring module.

Circuit Breaker State Transitions in Caddy #

Caddy’s Circuit Breaker operates as a three-state state machine: Closed, Open, and Half-Open.

stateDiagram-v2
    [*] --> Closed: Start / Upstream Healthy
    
    Closed --> Open: "Accumulated Failures >= max_fails\n(within the fail_duration window)"
    note right of Closed
        NORMAL status.
        Traffic is sent to all
        backends. Failures are tracked.
    end note

    Open --> HalfOpen: "The fail_duration Time\nhas Expired"
    note left of Open
        ERROR / ISOLATION status.
        Problematic backends are skipped.
        No traffic goes there.
    end note

    HalfOpen --> Closed: "First Request Succeeds\n(Back to Healthy Status)"
    HalfOpen --> Open: "First Request Fails\n(Re-isolation)"
    note right of HalfOpen
        TRIAL status.
        Caddy sends the next user
        request to this backend.
    end note

Here’s a detailed explanation of each state transition in the diagram above:

1. CLOSED Status (Normal State) #

In this state, the “circuit” is closed, meaning traffic flows smoothly from Caddy to all backend servers. Caddy actively monitors backend responses and records failures. As long as the failure count stays below the max_fails threshold, the status remains CLOSED.

2. OPEN Status (Disconnected / Isolation State) #

When a backend’s failure count reaches or exceeds max_fails within the fail_duration window, the circuit switches to OPEN status.

In this state, Caddy “cuts off” the flow to that backend. Caddy immediately isolates the backend server and won’t route user requests to it. This step gives your overwhelmed backend server breathing room to recover (e.g., auto-restart, clear memory, or finish piling database query queues) without being continuously bombarded by new requests.

3. HALF-OPEN Status (Trial State) #

After the fail_duration isolation time passes, the circuit automatically switches to HALF-OPEN status.

In this state, Caddy acts cautiously. Caddy doesn’t immediately mark the backend fully healthy. When the next user request arrives at Caddy, Caddy tries testing the backend by sending that request to it.

• If the trial request succeeds, the circuit returns to CLOSED status (fully healthy), clearing all previous failure records and restoring the backend to full rotation.
• If the trial request fails, the circuit immediately returns to OPEN status (disconnected), and Caddy re-isolates the backend for a new fail_duration.

Synergy with Retry Logic #

One of the main weaknesses of standalone passive monitoring is the side effect on end users. Because Caddy needs real traffic to detect failures, users who send requests when a backend first dies will receive error messages (like 502 Bad Gateway or 504 Gateway Timeout) before Caddy realizes the backend is dead and isolates it.

To overcome this weakness, you must combine Passive Health Check with Retry Logic using the lb_try_duration and lb_try_interval directives.

How the Passive and Retry Synergy Works #

When you enable retry, Caddy transparently shifts traffic in the background when it detects connection failures:

1. Request Arrives: A user sends a request to Caddy.
2. First Attempt: Caddy routes the request to backend-1, which just died.
3. Failure Detection: Caddy detects the connection to backend-1 is refused (Connection Refused).
4. Passive Recording: The passive health check records 1 failure for backend-1.
5. Retry Activation: Caddy doesn’t immediately return an error to the user. Caddy sees that lb_try_duration is configured.
6. Second Attempt (Retry): Caddy immediately shifts the same request to the healthy backend-2.
7. Transparent Success: backend-2 responds successfully. The user receives the web page normally without realizing a failure briefly happened on the main backend.

Let’s look at the visualization of this interaction through the following sequence diagram:

sequenceDiagram
    participant Client as Web Visitor
    participant Caddy as Caddy Gateway
    participant B1 as Backend 1 (Just Died)
    participant B2 as Backend 2 (Healthy)

    Client->>Caddy: Send HTTP Request
    Caddy->>B1: Forward Request (First Attempt)
    Note over B1: B1 experiences an internal crash
    B1-->>Caddy: Connection Refused / Error
    Note over Caddy: Caddy records a passive failure for B1 (+1 Fail Counter)
    
    Note over Caddy: Caddy retries because lb_try_duration is active
    Caddy->>B2: Forward Request to the Alternative Backend
    B2-->>Caddy: HTTP/1.1 200 OK (Success)
    Caddy-->>Client: Return Successful Response
    Note over Client,B2: The visitor doesn't feel any error at all!

Production Configuration for the Passive and Retry Synergy #

Here’s an example Caddyfile configuration optimally combining passive monitoring, latency limits, and retry logic for production environments:

# High resilience configuration in production
api.example.com {
    reverse_proxy backend-1:8080 backend-2:8080 backend-3:8080 {
        # ═══ Passive Monitoring (Circuit Breaker) ═══
        # Enable passive tracking with 30-second isolation
        fail_duration 30s
        # Isolate a backend after 2 failures within the 30s window
        max_fails 2
        # Consider statuses 502, 503, and 504 as application failures
        unhealthy_status 502 503 504
        # Consider responses slower than 3 seconds as failures
        unhealthy_latency 3s

        # ═══ Retry Logic (Transparent Fault Tolerance) ═══
        # Caddy keeps trying other backends for up to 5 seconds
        lb_try_duration 5s
        # Wait 200ms before trying the next backend
        lb_try_interval 200ms
        
        # HTTP transport configuration to speed up failure detection
        transport http {
            dial_timeout 2s
            response_header_timeout 5s
        }
    }
}

The Best Combination: Active + Passive Health Check #

To build a truly reliable load balancing infrastructure (High Availability), you must not choose between active or passive monitoring. You must combine both so they complement each other.

Monitoring Behavior Comparison Table #

Layered Defense Architecture (Unified Checking) #

When you use both methods simultaneously, you create a layered defense system:

1. First Layer (Active Check - Prevention): Every 10 seconds, Caddy’s poller checks all backends’ health. If a backend server dies in the early morning (when there’s no user traffic), the active health check detects it and isolates it immediately before the first user wakes up and accesses your application.
2. Second Layer (Passive Check - Real-Time Protection): If a backend server suddenly runs out of RAM (Out Of Memory) or gets a locked database query (database lock) right after passing the 5th active check scan, the next user request that fails is immediately caught by the passive monitor. Caddy instantly isolates that server without waiting for the next active check schedule.

Here’s the complete layered defense configuration in the Caddyfile:

# Layered defense configuration for critical applications
app.example.com {
    reverse_proxy backend-1:8080 backend-2:8080 backend-3:8080 {
        # Use the least_conn policy so the load splits fairly
        lb_policy least_conn

        # ─── LAYER 1: Active Health Check (Background Monitoring) ───
        health_uri /healthz
        health_interval 15s
        health_timeout 3s
        # Requires an authentication token to keep the endpoint secure
        health_headers {
            Authorization "Bearer caddy-internal-security-token"
        }

        # ─── LAYER 2: Passive Health Check (Real Traffic Monitoring) ───
        fail_duration 60s
        max_fails 3
        unhealthy_status 500 502 503 504
        unhealthy_latency 4s

        # ─── LAYER 3: Retry Logic (User Request Rescuer) ───
        lb_try_duration 8s
        lb_try_interval 250ms

        # Transport-level configuration so timeouts are well managed
        transport http {
            dial_timeout 3s
            response_header_timeout 10s
        }
    }
}

Configuration for Various Scenarios #

Every application has different characteristics and failure tolerances. You must adjust passive monitoring parameters based on the field scenario’s needs:

1. Non-Critical Service Scenario / Loose Tolerance (Loose Config) #

For internal services, static documentation, or testing servers (staging) where 100% availability isn’t heavily demanded and you want to save CPU resources from evaluating failures too often:

# Loose tolerance configuration for non-critical services
staging.example.com {
    reverse_proxy staging-1:3000 staging-2:3000 {
        # A fairly short isolation time
        fail_duration 15s
        # Only isolate after a fairly massive failure
        max_fails 5
        # Only detect basic network failures, ignore slow latency
    }
}

2. Critical Service Scenario / Very Strict Tolerance (Strict Config) #

For banking applications, payment gateways, transaction APIs, or e-commerce services where a single mistake can disrupt business operations and user comfort:

# Super strict configuration for financial transactions
checkout.example.com {
    reverse_proxy prod-1:9000 prod-2:9000 prod-3:9000 {
        lb_policy least_conn
        
        # A fairly long isolation to ensure the backend truly recovers
        fail_duration 2m
        # Immediately isolate a backend after 1 failure!
        max_fails 1
        # Very sensitive to internal server errors
        unhealthy_status 500 502 503 504
        # Very sensitive to system slowness
        unhealthy_latency 1.5s
        
        # Very aggressive retry to rescue user requests
        lb_try_duration 4s
        lb_try_interval 100ms
        
        transport http {
            dial_timeout 1s
            response_header_timeout 2s
        }
    }
}

Logging, Observability, and Troubleshooting #

To ensure your passive monitoring works correctly and to transparently monitor backend cluster health, you must configure Caddy’s logging adequately and learn how to analyze failure events from those log files.

Configuring Caddy Access Logs #

You should enable JSON-format log writing at the global level so access and load balancing data can be easily read by log collection applications (like ElasticSearch, Loki, or Datadog):

# Global log configuration in the Caddyfile
{
    log {
        output file /var/log/caddy/system.log {
            roll_size 100mib
            roll_keep 10
        }
        format json
    }
}

api.example.com {
    log {
        output file /var/log/caddy/api_access.log
        format json
    }
    reverse_proxy backend-1:8080 backend-2:8080 {
        fail_duration 30s
        max_fails 2
    }
}

Analyzing Passive Monitoring Event Logs #

When Caddy detects failures and decides to isolate a backend server, Caddy writes a system log at the INFO security level.

Example Log: Upstream Isolated (Unhealthy) #

Here’s an example of the JSON log data structure when a backend is marked unavailable for exceeding max_fails:

{
  "level": "info",
  "ts": 1781682415.123456,
  "logger": "http.handlers.reverse_proxy",
  "msg": "upstream is now unavailable",
  "upstream": "backend-1:8080",
  "total_fails": 2
}

Example Log: Upstream Healthy Again #

When the fail_duration expires and the backend successfully processes the first trial request in HALF-OPEN status, Caddy writes the following recovery log:

{
  "level": "info",
  "ts": 1781682445.654321,
  "logger": "http.handlers.reverse_proxy",
  "msg": "upstream is now available",
  "upstream": "backend-1:8080"
}

Terminal Commands for Quick Analysis (CLI Debugging) #

As a system administrator, you can monitor these events directly on the server using standard Unix commands.

1. Real-Time Log Filtering for Backend Health Events #

Run the following command to watch backend status changes directly in your terminal:

# Monitor upstream status changes in real time from the Caddy system log
tail -f /var/log/caddy/system.log | grep -E --line-buffered "upstream is now"

# Example Output:
# {"level":"info","ts":1781682415.123,"logger":"http.handlers.reverse_proxy","msg":"upstream is now unavailable","upstream":"10.0.1.15:8080","total_fails":2}
# {"level":"info","ts":1781682445.654,"logger":"http.handlers.reverse_proxy","msg":"upstream is now available","upstream":"10.0.1.15:8080"}

2. Calculating the Error Ratio per Upstream Using jq and awk #

If you want to analyze the access log file to see the error percentage generated by each upstream address within a certain period, run this one-liner:

# Extract upstream_addr and status code data from the JSON log, then calculate the error ratio
cat /var/log/caddy/api_access.log | jq -r 'select(.upstream_addr != null) | [.upstream_addr, (.status | tostring)] | @tsv' | \
awk -F'\t' '{
    total[$1]++
    if ($2 >= "500") errors[$1]++
} END {
    print "=== UPSTREAM ERROR RATIO REPORT ==="
    for (addr in total) {
        err_rate = (errors[addr] ? errors[addr] : 0) / total[addr] * 100
        printf "%s -> Total Requests: %d, Errors (5xx): %d, Error Ratio: %.2f%%\n", addr, total[addr], (errors[addr] ? errors[addr] : 0), err_rate
    }
}'

The command above gives a structured report output like this:

=== UPSTREAM ERROR RATIO REPORT ===
10.0.1.15:8080 -> Total Requests: 14205, Errors (5xx): 14, Error Ratio: 0.10%
10.0.1.16:8080 -> Total Requests: 14198, Errors (5xx): 348, Error Ratio: 2.45%

With this report, you can immediately suspect that the 10.0.1.16 server is experiencing internal stability problems because its error ratio is much higher than its counterpart.

Implementation Review Checklist #

To ensure your Passive Health Check configuration runs safely and efficiently in production, check off all the following points before releasing:

CATEGORY 1: PARAMETER CONFIGURATION
  □ fail_duration is set to a value > 0 (passive monitoring won't activate if it's 0).
  □ max_fails is adjusted to your cluster's redundancy level (2-3 is recommended to avoid false alarms).
  □ unhealthy_status only contains server status codes (5xx) and does NOT include client status codes (4xx).
  □ unhealthy_latency is set to avoid request queue pile-ups on slow backends.

CATEGORY 2: RESILIENCE AND RETRY LOGIC
  □ lb_try_duration is enabled to ensure user requests are shifted to other backends during passive failures.
  □ The lb_try_duration value is longer than dial_timeout in the transport block.
  □ The retry pause (lb_try_interval) is set rationally to avoid flooding alternative backends.

CATEGORY 3: MONITORING INTEGRATION
  □ Active health checks are configured together with passive health checks for optimal layered protection.
  □ The global log format is set to JSON so it's easily mapped by external log collection systems.

Summary #

• Key Definition: The Passive Health Check feature observes status and responses from real user traffic to backend servers without generating artificial network traffic.
• Feature Activation: You must set fail_duration to a duration value greater than zero for the passive monitoring feature to operate.
• Circuit Breaker Pattern: Caddy automatically applies the circuit breaker pattern with three states (Closed, Open, Half-Open) to isolate problematic backend servers.
• Application Detection: Use the unhealthy_status and unhealthy_latency subdirectives to filter slow-running backends or ones returning internal server errors.
• Retry Synergy: Combine passive monitoring with lb_try_duration so the first failure on a broken backend is transparently shifted to an alternative backend without surfacing errors to the user.
• Layered Defense: Integrate active (proactive) and passive (reactive) monitoring simultaneously to guarantee backend cluster reliability at the production level.

← Previous: Active Health Check   Next: Admin API →