Least Connections #

Distributing web traffic evenly across a server cluster isn’t always enough when every request has a different computational load. This is where the Least Connections algorithm (least_conn) becomes the top choice over plain Round Robin. By monitoring the real-time active connection count on each backend server, Caddy can intelligently route new traffic to the server with the lightest current workload. We’ll thoroughly examine how this algorithm works, heterogeneous backend handling, mitigating the initial surge problem (slow-start), and performance comparisons under real workloads.

How the Least Connections Algorithm Works #

The Least Connections algorithm is based on the principle of actual workload fairness, not just transaction count fairness. Caddy maintains an internal gauge counter to track the number of active requests each backend (upstream) server is processing at any given time.

Internally in Caddy’s Go source code, every upstream is represented as a structure (Struct) object recording the dial address, health status, and active connection counter atomically:

// Internal Upstream representation in Caddy's Go source code
type Upstream struct {
    Dial        string
    Healthy     bool
    Conns       int32  // Its value is changed using sync/atomic thread-safely
}

When a new request arrives from a client:

1. Caddy checks the gauge counter (Conns) status of all registered healthy upstreams (Healthy == true).
2. Caddy compares the active connection counts among them.
3. The new request is routed to the upstream with the smallest active connection count.
4. Caddy increments the gauge counter for the selected upstream by +1 atomically (atomic.AddInt32(&upstream.Conns, 1)).
5. After the backend finishes processing the request and sends the response back, Caddy decrements that backend’s gauge counter by -1 (atomic.AddInt32(&upstream.Conns, -1)).

Here’s a simplified view of the upstream selection logic inside Caddy:

// Illustration of the Least Connections selection algorithm in Caddy
func SelectLeastConn(upstreams []*Upstream) *Upstream {
    var best *Upstream
    var minConns int32 = -1

    for _, upstream := range upstreams {
        if !upstream.Healthy {
            continue
        }
        
        // Read the connection count atomically for concurrency safety
        currentConns := atomic.LoadInt32(&upstream.Conns)
        
        if minConns == -1 || currentConns < minConns {
            minConns = currentConns
            best = upstream
        }
    }
    return best
}

Here’s a comparison diagram between Round Robin and Least Connections when facing requests with varying processing durations:

flowchart TD
    subgraph Round_Robin_Flow["Blind Rotation (Round Robin)"]
        direction TB
        R1["Req 1 (Fast: 50ms)"] -->|"Rotation"| RR1["Server 1 (Load: 1 conn)"]
        R2["Req 2 (Slow: 5s)"] -->|"Rotation"| RR2["Server 2 (Load: 3 conn - Overwhelmed!)"]
        R3["Req 3 (Slow: 5s)"] -->|"Rotation"| RR2
        R4["Req 4 (Slow: 5s)"] -->|"Rotation"| RR2
    end

    subgraph Least_Conn_Flow["Smart Balancing (Least Connections)"]
        direction TB
        L1["Req 1 (Fast: 50ms)"] -->|"Load Evaluation"| LC1["Server 1 (Load: 1 conn)"]
        L2["Req 2 (Slow: 5s)"] -->|"Load Evaluation"| LC2["Server 2 (Load: 1 conn)"]
        L3["Req 3 (Slow: 5s)"] -->|"Send to Server 1 because load is 0"| LC1
        L4["Req 4 (Slow: 5s)"] -->|"Send to Server 1 because 1 vs 2"| LC1
    end

    style RR2 stroke:#e53935,stroke-width:2px
    style LC1 stroke:#43a047,stroke-width:2px
    style LC2 stroke:#43a047,stroke-width:2px

In the Round Robin scenario (left), Server 2 becomes overwhelmed because it keeps receiving new requests without any regard for the fact that Server 2 is still busy processing previous slow requests. In contrast, Least Connections (right) detects that Server 2 is busy and routes new requests to Server 1, which has more spare capacity.

Least Connections Configuration #

To enable the Least Connections policy in Caddy, you must define the lb_policy least_conn subdirective inside your Caddyfile’s reverse_proxy block:

# Configuring the Least Connections load balancer
example.com {
    reverse_proxy backend-1:3000 backend-2:3000 backend-3:3000 {
        # Set the load balancing policy to Least Connections
        lb_policy least_conn
        
        # Connection failure tolerance configuration
        lb_try_duration 5s
        lb_try_interval 200ms
        
        # TCP transport settings to the backend
        transport http {
            dial_timeout 3s
            keep_alive 90s
        }
    }
}

• lb_policy least_conn: Instructs Caddy’s proxy module to evaluate the active connection count before routing requests.
• dial_timeout 3s: Ensures Caddy drops the connection quickly if the selected backend fails to respond on the TCP socket within 3 seconds, so the request can be shifted to the next lowest-connection backend.

The Slow-Start Mechanism (Thundering Herd Mitigation) #

Although Least Connections is very effective at balancing load, it has one critical weakness when you add a new backend server to the cluster (whether due to auto-scaling or after a server recovers from a crash). This problem is known as the Thundering Herd or Cold Start Problem.

The Thundering Herd Problem with Least Connections #

Imagine you have 2 active servers (Server A (100 conn) and Server B (100 conn)) busy processing active connections. When Server C just starts with 0 active connections, all new incoming requests arriving at the same time will be routed entirely to Server C because its value is still far below Servers A and B. The freshly started Server C, whose memory isn’t fully warmed up (cold memory), will instantly collapse under the extreme load surge.

Design Solution: Proxy Queuing via max_conns_per_host and lb_try_duration #

To mitigate the thundering herd on Least Connections in Caddy, you can apply a safe queuing pattern (connection queuing). You limit the maximum active connection capacity per host using max_conns_per_host at the transport level, combined with lb_try_duration at the proxy level:

# Safe queuing to dampen the surge on new backends
example.com {
    reverse_proxy backend-1:3000 backend-2:3000 backend-3:3000 {
        lb_policy least_conn
        
        # Give requests time to queue in Caddy rather than immediately erroring with 503
        lb_try_duration 10s
        lb_try_interval 100ms
        
        transport http {
            # Limit the maximum active connections to each backend
            max_conns_per_host 120
            
            dial_timeout 3s
        }
    }
}

With the configuration above:

1. When the newly recovered Server C receives an instant overflow of requests from 0 to 120 connections, Caddy detects that Server C has hit the maximum max_conns_per_host 120 limit.
2. Request #121 isn’t sent to Server C. It’s held in Caddy’s queue (proxy buffer) for the lb_try_duration 10s duration.
3. During the wait, if one active connection on Server C finishes processing, queued request #121 immediately takes its place in an orderly manner.
4. This gives Server C room to process requests in a controlled way without collapsing.

The Thundering Herd Problem with Mass WebSocket Reconnections #

The thundering herd scenario above becomes very dangerous if your cluster serves long-lived persistent connections like WebSockets or gRPC streams.

1. Server C (:3003) is restarted for routine maintenance.
2. The 2,000 WebSocket clients connected to Server C disconnect instantly.
3. Those clients immediately send automatic reconnection requests at the same time.
4. Because Server C just came back up with a 0-connection load, Least Connections consistently throws ALL 2,000 reconnection requests back to Server C.
5. Server C collapses at the CPU level before it can even complete the SSL/TLS handshakes.

Client Solution: Adding Jitter and Exponential Backoff #

The best solution for this problem is inserting random time variation (Jitter) into the automatic reconnection logic on your client application code side (JavaScript browser):

// ANTI-PATTERN: Instant reconnection that triggers the thundering herd
socket.onclose = () => {
    setTimeout(connectWebSocket, 1000); // All clients reconnect in exactly 1 second
};

// CORRECT: Using Exponential Backoff with random Jitter
socket.onclose = () => {
    // Add a random factor between 1 and 5 seconds
    const jitter = Math.random() * 4000 + 1000;
    setTimeout(connectWebSocket, jitter);
};

Comparative Analysis: When to Choose the Right Algorithm #

Here’s a performance comparison matrix between Round Robin, Least Connections, and IP Hashing across various traffic load types:

Simulation & Performance Analysis #

To understand why Least Connections excels at handling heterogeneous traffic, let’s look at the request simulation table below.

Simulation Scenario #

We have 2 backend servers:

• Server A (Fast VM): Processes regular CRUD database requests in 10ms.
• Server B (Slow VM): Processes heavy PDF file creation requests in 100ms.

We send 5 incoming requests simultaneously at time T = 0ms:

• Request 1: Light Category
• Request 2: Heavy Category
• Request 3: Light Category
• Request 4: Light Category
• Request 5: Light Category

Request Progression Results with Round Robin (Default) #

• Problem: Request 4 (light) is forced to queue behind Request 2 on the slow Server B. The user on Request 4 must wait 200ms even though their request is very light, while Server A has been idle since T=30ms.

Request Progression Results with Least Connections #

• Result: Request 4 is intelligently routed by Caddy to Server A, which has fast processing capacity. The Request 4 user receives the response in 30ms (far faster than the 200ms on Round Robin). Server B is left focused on processing its one heavy task without being disturbed by new request queues.

Supporting Backend Code Implementation #

To independently test Caddy’s Least Connections load balancer performance on your local computer, you can create a simulation backend application using the Python programming language. The code below deliberately simulates heavy tasks with a dynamic sleep delay parameter:

# app.py (Simulation Backend Application via Python Flask)
import os
import time
import sys
from flask import Flask, jsonify, request

app = Flask(__name__)

# Read the instance name configuration from the environment variable
INSTANCE_NAME = os.getenv("INSTANCE_NAME", "Backend-Default")
PORT = int(os.getenv("PORT", 5000))

# Local counter to record the number of processed requests
request_counter = 0

@app.route("/api/data")
def process_data():
    global request_counter
    request_counter += 1
    
    # Get the delay parameter from the query string (default: 0 seconds)
    # Example: /api/data?sleep=5 (simulates a 5-second heavy task)
    sleep_duration = float(request.args.get("sleep", 0))
    
    print(f"[{INSTANCE_NAME}] Received request #{request_counter}. Processing duration: {sleep_duration}s", file=sys.stderr)
    
    if sleep_duration > 0:
        time.sleep(sleep_duration)
        
    return jsonify({
        "status": "success",
        "instance": INSTANCE_NAME,
        "processed_requests": request_counter,
        "slept_for": sleep_duration
    })

@app.route("/healthz")
def health_check():
    return jsonify({"status": "UP"}), 200

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=PORT)

Docker Compose Configuration for the Simulation #

To run this test end to end, create the following docker-compose.yml file:

version: "3"
services:
  caddy:
    image: caddy:2.8-alpine
    ports:
      - "80:80"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
    networks:
      - net

  backend-a:
    build: .
    environment:
      - INSTANCE_NAME=Server-A
      - PORT=5000
    networks:
      - net

  backend-b:
    build: .
    environment:
      - INSTANCE_NAME=Server-B
      - PORT=5000
    networks:
      - net

networks:
  net:

Use the Apache Bench (ab) command below to send concurrent requests and observe the Docker container logs to see how Caddy intelligently sorts traffic:

# Send 100 requests with 10 concurrent connections to Caddy
ab -n 100 -c 10 http://localhost/api/data?sleep=1

Summary #

• Key Definition: The Least Connections algorithm selects the backend server with the fewest active connections when a new request arrives.
• Connection Gauge: Caddy maintains a real-time numeric counter in memory to detect when backends start getting busy processing heavy computational tasks.
• Heterogeneous Backends: It’s the best choice for server clusters with varying CPU/RAM capacities, and for requests with non-homogeneous response times.
• Thundering Herd Danger: A newly started server risks collapsing from receiving accumulated instant requests because its connection count is still zero.
• Thundering Herd Solution: Set the max_conns_per_host limit in the transport section to cap the initial load surge on newly activated servers.
• Ideal Cases: Highly recommended for large file upload applications, heavy query APIs, and persistent connection services like WebSocket and gRPC.

← Previous: Round Robin   Next: IP Hash →