Active Health Check #

Making sure a load balancer only routes traffic to servers genuinely ready to process data is an absolute prerequisite for maintaining system stability. Caddy provides the Active Health Check feature, which works proactively by testing backend server reliability before real users send their requests. We’ll break down Caddy’s internal monitoring engine architecture, all configuration parameters (like HTTP status and body substring validation), methods for securing health endpoints from external exploitation, and strategies for designing multi-resource health detectors at the application level.

How Active Health Check Works #

The Active Health Check mechanism works on the proactive and periodic principle. Caddy periodically sends artificial HTTP requests (probes) to a special path you define on each backend (upstream) server.

This monitoring process runs independently in the background:

  1. Background Worker Goroutines: Caddy creates a dedicated background goroutine acting as a health poller. This goroutine runs asynchronously without disturbing the real user traffic processing flow.
  2. Send HTTP Probe: Each time the interval duration passes (e.g., every 10 seconds), the poller sends an HTTP request (usually a GET or HEAD method) to the backend.
  3. Validate Response: The poller receives the response from the backend and tests it against the health eligibility criteria you set (HTTP status code and response body contents).
  4. Atomic Flag Update: If the backend passes the criteria, the poller marks that backend as healthy (Healthy = true). If it fails (or hits a connection timeout), the poller immediately marks it as unhealthy (Healthy = false). This status update is done atomically in memory so the main load balancing goroutine can read it instantly.

In-Memory Upstream Consolidation (De-duplication) #

In large architectures where you define dozens of different domain names (virtual hosts) pointing at the same backend cluster, Caddy intelligently consolidates its in-memory upstream pointer registry. If both api.example.com and app.example.com have the backend 10.0.1.15:3000, Caddy doesn’t create two separate poller goroutines for that backend. Caddy de-duplicates the registration and only runs a single poller daemon to test that server’s health. This saves proxy server RAM usage and prevents flooding your backend servers with request logs.

Here’s the internal architecture of Caddy’s active health check engine:

flowchart TD
    subgraph Caddy_Proxy_Process["Main Caddy Process"]
        LB["Load Balancer Goroutine"]
        Poller["Active Poller Goroutine (Background)"]
    end

    subgraph Backend_Servers["Backend Cluster"]
        B1["Server 1 (Status: Healthy)"]
        B2["Server 2 (Status: Unhealthy)"]
    end

    Poller -->|"1. Send Periodic HTTP Probe"| B1
    Poller -->|"2. Send Periodic HTTP Probe"| B2
    
    B1 -->|"3. Return 200 OK"| Poller
    B2 -->|"4. Connection Timeout / 503"| Poller
    
    Poller -->|"5. Update atomic flag Healthy=true"| B1
    Poller -->|"6. Update atomic flag Healthy=false"| B2
    
    LB -->|"7. Only route requests to Server 1"| B1

    style Poller stroke:#0288d1,stroke-width:2px
    style B1 stroke:#43a047,stroke-width:2px
    style B2 stroke:#e53935,stroke-dasharray: 5,5

With this proactive architecture, if Server 2 crashes at 10:00:00, Caddy detects it at 10:00:10. When a user sends a request at 10:00:11, Caddy skips Server 2 immediately without trying to connect to it first, sparing users the bad experience of a slow connection error.

Active Health Check Parameter Configuration #

The Caddyfile provides a set of very flexible configuration parameters for adjusting active monitoring behavior inside the reverse_proxy block:

# Complete active monitoring configuration
example.com {
    reverse_proxy backend-1:3000 backend-2:3000 {
        # ═══ Active Health Check Parameters ═══
        # The health URL path on the backend
        health_uri /healthz
        
        # Pause between checks (default: 30 seconds)
        health_interval 10s
        
        # Probe response wait limit
        health_timeout 3s
        
        # HTTP status code considered healthy
        health_status 200
        
        # Send custom headers when probing
        health_headers {
            X-Health-Checker "Caddy-Gateway"
            Authorization "Bearer local-monitor-secret-token"
        }
        
        # Specify a special port for health probes (optional)
        # health_port 8081
    }
}
  • health_uri: The backend endpoint for checking (e.g., /healthz, /status, or /ping).
  • health_interval: Controls how often the poller sends probes. Setting the interval too fast (like 1s) burdens the backend’s logs and resources. Setting it too slow (like 1m) slows down failure detection. The ideal value is 5s to 15s.
  • health_timeout: The socket response tolerance limit. If the backend is processing extreme load so the TCP/HTTP handshake is delayed past 3 seconds, Caddy considers the backend sick so it doesn’t slow down user traffic.
  • health_headers: Very useful if your backend sits behind an application firewall requiring special authentication headers, or for recording logger identity on the backend.

Advanced Health Validation (Response Body Matching) #

Judging backend server health only by the 200 OK HTTP status code risks the system being tricked. Often, web applications with broken internal database connections or exhausted RAM still process HTTP requests normally and return 200 OK status — but with an error message JSON payload in the body:

// ANTI-PATTERN: HTTP 200 OK status, but the internal application is broken!
{
  "status": "error",
  "error": "connection to PostgreSQL pool timed out"
}

If Caddy only validates the 200 status, it thinks the backend is healthy and keeps sending user requests there, serving JSON error pages to site visitors.

To overcome this, you can use the health_body subdirective to validate the payload contents returned by the backend:

# Response body substring validation
example.com {
    reverse_proxy backend-1:3000 backend-2:3000 {
        health_uri /healthz
        health_status 200
        
        # Caddy only declares the backend healthy if the response body contains the string below
        health_body "\"status\":\"UP\""
    }
}

Caddy’s poller reads the response body byte stream and searches for that substring match. If the HTTP status is 200 but the returned body contains an error message, the backend is immediately declared sick and automatically disabled.

Health Endpoint Security #

Leaving health endpoints like /healthz openly exposed to the public internet carries significant security risks:

  1. Sensitive Information Leakage: Details like database status, framework versions, server memory, and internal cluster dependencies can be freely read by attackers.
  2. Denial of Service (DDoS) Attacks: A /healthz endpoint running heavy database queries can be exploited by attackers to flood the server with consecutive requests and stall your database.

Here are the three best strategies for securing health endpoints:

1. Use a Separate Port (health_port) #

Configure your backend server to serve normal application traffic routes on the main port (e.g., :3000) and serve the /healthz route on a special administrative port (e.g., :8081).

Then, close port :8081 from external access using OS firewall rules (cloud Security Group or Linux UFW), and instruct Caddy to probe that port:

# Point probes at the special private port
example.com {
    reverse_proxy backend-1:3000 backend-2:3000 {
        health_uri /healthz
        
        # Send health probes to the backend's private port 8081
        health_port 8081
    }
}

2. Authentication Token Validation (health_headers) #

Instruct Caddy to include a unique authentication token in the HTTP probe header, and configure your backend code to refuse access if the token doesn’t match:

# Send a monitor token
example.com {
    reverse_proxy backend-1:3000 backend-2:3000 {
        health_uri /healthz
        health_headers {
            Authorization "Bearer local-monitor-token-123"
        }
    }
}

3. Security Using Mutual TLS (mTLS) #

If your TLS backend requires client certificate validation, Caddy automatically uses the HTTP transport configuration defined on the relevant reverse proxy to perform health probes. That means the mTLS certificates you register inside the transport http block are automatically sent by Caddy’s background poller when testing /healthz on your HTTPS backend:

# Integrated mTLS authentication for health probes
example.com {
    reverse_proxy https://secure-backend:8443 {
        health_uri /healthz
        
        transport http {
            tls
            tls_client_auth /etc/certs/caddy.crt /etc/certs/caddy.key
            tls_trusted_ca_certs /etc/certs/root-ca.crt
        }
    }
}

The Danger of Cascading Failure from Wrong Configuration #

Cascading Failure is a terrifying specter for infrastructure architects. Misconfiguring active monitoring timing can accidentally accelerate the death of every server in your cluster during a traffic spike.

The Cluster Collapse Incident Scenario #

Imagine you have 3 backend servers with a very strict active monitoring setup: health_interval 2s and health_timeout 1s.

1. Server 1 suffers a hardware failure and dies.
2. The traffic load carried by Server 1 is entirely shifted to Server 2 and Server 3.
3. The load on Servers 2 and 3 spikes dramatically, pushing their CPU utilization to 100%.
4. Due to the 100% CPU workload, Servers 2 and 3's response times slow from 50ms to 1.2 seconds.
5. At the same time, Caddy's poller sends health probes. Because the backend response time (1.2 seconds) exceeds the "health_timeout 1s" limit, the poller considers Servers 2 and 3 dead.
6. Caddy instantly closes all traffic flow to Servers 2 and 3.
7. The entire cluster goes completely dark (503 Service Unavailable) for all users, even though Servers 2 and 3 are actually still alive and struggling to process requests at maximum capacity.

Safe Design Recommendations #

To prevent this disaster:

  • Don’t set health_timeout too tight. The timeout value must cover the backend’s worst-case latency under high load (e.g., 3s or 5s).
  • Set health_interval rationally (e.g., 10s or 15s) to reduce the extra computational burden on already-overwhelmed backends.

Multi-Resource Health Monitoring Strategy (Backend Code) #

A reliable health endpoint at the application level should test the readiness of all its critical dependencies. Look at this advanced /healthz endpoint implementation example using the Go programming language, monitoring SQL database connectivity, Redis cache, and local storage capacity:

package main

import (
	"database/sql"
	"encoding/json"
	"net/http"
	"syscall"
	"time"

	"github.com/go-redis/redis/v8"
	_ "github.com/lib/pq"
)

type HealthResponse struct {
	Status    string            `json:"status"`
	Timestamp string            `json:"timestamp"`
	Resources map[string]string `json:"resources"`
}

var (
	db    *sql.DB
	rdb   *redis.Client
)

func healthHandler(w http.ResponseWriter, r *http.Request) {
	// Initialize the default status
	response := HealthResponse{
		Status:    "UP",
		Timestamp: time.Now().Format(time.RFC3339),
		Resources: make(map[string]string),
	}

	// 1. Check the token authentication from Caddy
	token := r.Header.Get("Authorization")
	if token != "Bearer local-monitor-token-123" {
		http.Error(w, "Unauthorized Checker", http.StatusUnauthorized)
		return
	}

	// 2. Test SQL database connectivity (PostgreSQL)
	// We use Ping() instead of SELECT 1 because Go's Ping() has been
	// optimized to quickly validate the internal connection pool status
	// without forcing the database parser to compile an sql query.
	if err := db.Ping(); err != nil {
		response.Status = "DOWN"
		response.Resources["database"] = "DOWN: " + err.Error()
	} else {
		response.Resources["database"] = "UP"
	}

	// 3. Test Cache Server connectivity (Redis)
	if err := rdb.Ping(r.Context()).Err(); err != nil {
		response.Status = "DOWN"
		response.Resources["redis"] = "DOWN: " + err.Error()
	} else {
		response.Resources["redis"] = "UP"
	}

	// 4. Test the operating system's Disk Space capacity (Disk Space Check)
	// Syscall.Statfs reads the OS filesystem metadata directly (Linux/macOS)
	// to instantly test the availability of remaining memory blocks.
	var stat syscall.Statfs_t
	if err := syscall.Statfs("/", &stat); err == nil {
		// Calculate the remaining storage space in bytes
		freeDiskBytes := stat.Bavail * uint64(stat.Bsize)
		if freeDiskBytes < 500*1024*1024 { // Minimum limit of 500MB
			response.Status = "DOWN"
			response.Resources["disk_space"] = "CRITICAL: Free disk is less than 500MB"
		} else {
			response.Resources["disk_space"] = "OK"
		}
	}

	// Determine the HTTP status based on the checks passed
	statusCode := http.StatusOK
	if response.Status == "DOWN" {
		statusCode = http.StatusServiceUnavailable
	}

	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(statusCode)
	json.NewEncoder(w).Encode(response)
}

func main() {
	// (DB and Redis connection initialization code goes here)
	http.HandleFunc("/healthz", healthHandler)
	http.ListenAndServe(":8081", nil)
}

The Go application above listens for health probes on the private :8081 port, secures the data by validating the authentication token from Caddy, performs deep testing on all critical database resources, and safely returns an HTTP 503 Service Unavailable status if a database connectivity failure is detected.

Summary #

  • Key Definition: The Active Health Check feature periodically sends test requests (probes) in the background using asynchronous goroutines to backend servers.
  • Proactive System: Instantly isolates dead backend servers before real user requests can be sent to them.
  • Response Body Detection: Use the health_body directive to validate response string contents (like JSON status) to avoid letting backends with HTTP 200 status but internal database failures slip through.
  • Endpoint Security: Protect health endpoints from public exploitation by separating them onto a closed administrative port (health_port) or using header authentication token verification (health_headers).
  • Disaster Mitigation: Avoid setting health_timeout too tight to prevent cascading failures when servers experience load spikes.
  • Validation Design: Design application-level health monitors (Go handlers) that structurally test the readiness of database dependencies, redis status, disk space, and backend memory.

← Previous: Weighted   Next: Passive Health Check →

About | Author | Content Scope | Editorial Policy | Privacy Policy | Disclaimer | Contact