Round Robin #

Providing even traffic load distribution across several application servers is a mandatory step in building systems with high availability and fault tolerance. As a load balancer, Caddy provides the Round Robin algorithm as its default policy for distributing traffic alternately and evenly. We’ll learn how this classic algorithm works, its concurrency handling mechanism at the OS level, its application in containerized clusters, and its important implications for stateful vs stateless architectures.

How the Round Robin Algorithm Works #

Conceptually, Round Robin works like a circular ring buffer. When a series of requests arrives from client browsers, Caddy routes them sequentially to the available backend (upstream) servers:

flowchart TD
    K1["Client Request 1"] --> CP1["Caddy Proxy"] --> SA["Upstream Server A (Port :3001)"]
    K2["Client Request 2"] --> CP2["Caddy Proxy"] --> SB["Upstream Server B (Port :3002)"]
    K3["Client Request 3"] --> CP3["Caddy Proxy"] --> SC["Upstream Server C (Port :3003)"]
    K4["Client Request 4"] --> CP4["Caddy Proxy"] --> SA2["Upstream Server A (Starts rotating again)"]

Here’s a visualization of the Round Robin circular flow in Caddy:

flowchart TD
    Req1["Request 1"] --> Caddy{"Caddy Proxy"}
    Req2["Request 2"] --> Caddy
    Req3["Request 3"] --> Caddy
    Req4["Request 4"] --> Caddy

    subgraph Circular_Buffer["Upstream Rotation (Circular Buffer)"]
        direction LR
        S1["Server 1 (:3001)"]
        S2["Server 2 (:3002)"]
        S3["Server 3 (:3003)"]
    end

    Caddy -->|"1. Send to Server 1"| S1
    Caddy -->|"2. Send to Server 2"| S2
    Caddy -->|"3. Send to Server 3"| S3
    Caddy -->|"4. Rotate to Server 1"| S1

    style S1 stroke:#0288d1,stroke-width:2px
    style S2 stroke:#0288d1,stroke-width:2px
    style S3 stroke:#0288d1,stroke-width:2px

Concurrency Without Mutex (Atomic Operations) #

On high-traffic servers, Caddy processes thousands of requests simultaneously (concurrently) through different goroutines. If Caddy used a traditional locking mechanism (Mutex Lock) to record which server index should receive the next request, the server would suffer slowdowns from lock contention between goroutines.

To prevent this performance problem, Caddy internally implements thread-safe rotation index tracking using atomic operations (sync/atomic in Go). Atomic operations work directly at the CPU instruction level, letting goroutines safely increment the index pointer counter without locking system threads.

flowchart TD
    G1["Goroutine Request A"] -->|"Atomic.AddUint32()"| Counter["Global Counter (Atomic)"]
    G2["Goroutine Request B"] -->|"Atomic.AddUint32()"| Counter
    G3["Goroutine Request C"] -->|"Atomic.AddUint32()"| Counter

    Counter -->|"Calculate Index: Counter % Upstream Count"| IndexCalculated["Target Upstream Index"]
    IndexCalculated --> Dispatch["Route to Backend Server"]

    style Counter stroke:#43a047,stroke-width:2px

Round Robin load distribution guarantees even traffic splitting in terms of quantity (number of requests). However, this algorithm doesn’t care how heavy the computation is for each request. If Server A receives a financial report query request taking 5 seconds, while Server B receives a small image file request finishing in 10 milliseconds, Server A will appear much busier even though the number of requests it received equals Server B’s.

Round Robin Configuration #

Because Round Robin is Caddy’s built-in (default) algorithm, you don’t have to define it explicitly in the Caddyfile. Just register all upstreams behind the reverse_proxy directive.

# Implicit configuration (Round Robin runs automatically)
example.com {
    reverse_proxy app-node-1:3000 app-node-2:3000 app-node-3:3000
}

However, to clarify infrastructure documentation for your DevOps team, writing this policy explicitly is good practice:

# Explicit configuration
example.com {
    reverse_proxy app-node-1:3000 app-node-2:3000 app-node-3:3000 {
        # State the load balancing policy in writing
        lb_policy round_robin
        
        # Recommended production supporting configuration
        lb_try_duration 5s
        lb_try_interval 250ms
    }
}

• lb_policy round_robin: Explicitly instructs Caddy to use the circular sequential distribution method.
• lb_try_duration: Gives Caddy a 5-second time tolerance to try moving the request to the next backend server if the first targeted backend fails to respond.

Cluster Deployment with Docker Compose #

In the modern containerized ecosystem, the most common deployment pattern is placing a Caddy container at the front line to distribute load across several replica application containers behind it.

Here’s a docker-compose.yml file configuration for creating a cluster of 3 NodeJS application replicas behind Caddy:

# docker-compose.yml
version: "3.8"

services:
  caddy-proxy:
    image: caddy:2.8-alpine
    container_name: caddy_gateway
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    networks:
      - internal_cluster
    depends_on:
      - node-app-1
      - node-app-2
      - node-app-3

  node-app-1:
    image: node:20-alpine
    container_name: backend_node_1
    working_dir: /app
    volumes:
      - ./app:/app
    command: node index.js
    environment:
      - PORT=3000
      - INSTANCE_NAME=Server-Node-A
    networks:
      - internal_cluster

  node-app-2:
    image: node:20-alpine
    container_name: backend_node_2
    working_dir: /app
    volumes:
      - ./app:/app
    command: node index.js
    environment:
      - PORT=3000
      - INSTANCE_NAME=Server-Node-B
    networks:
      - internal_cluster

  node-app-3:
    image: node:20-alpine
    container_name: backend_node_3
    working_dir: /app
    volumes:
      - ./app:/app
    command: node index.js
    environment:
      - PORT=3000
      - INSTANCE_NAME=Server-Node-C
    networks:
      - internal_cluster

networks:
  internal_cluster:
    driver: bridge

volumes:
  caddy_data:
  caddy_config:

Plus the supporting Caddyfile that routes traffic in a circular pattern:

# Caddyfile
{
    email [email protected]
}

app.example.com {
    encode gzip zstd
    
    # Connect to all three Docker containers using the bridge network internal DNS
    reverse_proxy node-app-1:3000 node-app-2:3000 node-app-3:3000 {
        lb_policy round_robin
        
        # Integrate with the active health check system so dead containers are skipped automatically
        health_uri /healthz
        health_interval 10s
        health_timeout 3s
    }
}

Dynamic Scaling #

One of the biggest advantages of Caddy’s architecture is its support for dynamic configuration via REST API without restarting the system process. This is very useful if you want to automate backend server auto-scaling.

Behind the scenes, Caddy stores all its configuration in a centralized JSON document format. When you use API commands, Caddy modifies the JSON document in RAM directly. To see the JSON representation of the running Caddyfile, you can execute the adaptation command:

# Convert the Caddyfile into Caddy's structured JSON format
caddy adapt --config /etc/caddy/Caddyfile --pretty

When your DevOps team adds a new backend server (e.g., node-app-4:3000) during peak hours, you don’t need to drop active user connections by restarting Caddy. Just send a new HTTP POST request to the Caddy admin port (:2019):

# 1. Check the current active upstream list from the Caddy Admin API
curl -s http://localhost:2019/config/apps/http/servers/srv0/routes | jq

# 2. Add a new upstream instantly to the load balancer list
curl -X POST http://localhost:2019/config/apps/http/servers/srv0/routes/0/handle/0/routes/0/handle/0/upstreams \
  -H "Content-Type: application/json" \
  -d '{"dial": "node-app-4:3000"}'

Caddy immediately inserts node-app-4:3000 into the Round Robin rotation circuit gracefully. This process happens entirely in RAM without dropping a single active connection.

Auto-scaler Integration Automation Script #

Here’s an example bash shell script that can be run periodically by Cron or an internal orchestration system to monitor CPU load and dynamically register backup containers to the Caddy load balancer:

#!/bin/bash
# autoscale_caddy.sh

THRESHOLD_CPU=80
CADDY_API="http://localhost:2019/config/apps/http/servers/srv0/routes/0/handle/0/routes/0/handle/0/upstreams"

# Get the current average CPU usage
CPU_USAGE=$(vmstat 1 2 | tail -1 | awk '{print 100 - $15}')

echo "Current CPU usage: $CPU_USAGE%"

if [ "$CPU_USAGE" -gt "$THRESHOLD_CPU" ]; then
    echo "High load detected! Starting an additional backend container..."
    
    # Run the new container via docker-compose
    docker compose up -d --no-recreate node-app-4
    
    # Wait 3 seconds for the new container to warm up
    sleep 3
    
    # CORRECT: Register it to the dynamic Caddy load balancer
    curl -s -X POST "$CADDY_API" \
      -H "Content-Type: application/json" \
      -d '{"dial": "node-app-4:3000"}' \
      -o /dev/null
      
    echo "node-app-4 successfully registered to Caddy!"
fi

Stateless vs Stateful Load Balancing #

When designing an application architecture behind a Round Robin load balancer, one absolute rule must be followed: your backend application must be stateless.

The Problem of Stateful Architecture in Round Robin #

If your backend application stores session data (session state) in each server’s local RAM, the following disaster scenario happens:

sequenceDiagram
    autonumber
    actor Client as Client Browser
    participant Caddy as Caddy Proxy
    participant ServerA as Server A (:3001)
    participant ServerB as Server B (:3002)

    Client->>Caddy: Send login request
    Caddy->>ServerA: Forward request (Round Robin)
    Note over ServerA: Process login & store<br>session data in local RAM
    ServerA-->>Client: Login Success

    Client->>Caddy: Request profile page (Click profile)
    Caddy->>ServerB: Forward request (Round Robin)
    Note over ServerB: Check local RAM...<br>Session data not found!
    ServerB-->>Client: Error "401 Unauthorized"

Solution 1: Change the Architecture to Stateless (Highly Recommended) #

Never store user session data (like PHP session files, Node.js sessions, or shopping carts) in the application server’s local memory. Move that session storage to a centralized database shared by all backends, like Redis or Memcached:

flowchart LR
    Client["Client"] --> Caddy["Caddy Proxy"]
    
    subgraph Backends["Backend Servers"]
        direction TB
        ServerA["Server A (:3001)"]
        ServerB["Server B (:3002)"]
        ServerC["Server C (:3003)"]
    end
    
    Caddy --> ServerA
    Caddy --> ServerB
    Caddy --> ServerC
    
    ServerA --> Redis["Redis Session Store"]
    ServerB --> Redis
    ServerC --> Redis

    style Caddy stroke:#0288d1,stroke-width:2px
    style Redis stroke:#43a047,stroke-width:2px
    style Backends stroke:#757575,stroke-width:1px,stroke-dasharray:5,5

With this stateless architecture, no matter where Caddy routes the client request (whether to Server A, B, or C), all backends can read the same session data from the shared Redis database.

Solution 2: Use Sticky Sessions (Alternative) #

If you’re forced to use a third-party application that doesn’t support external session storage, you must change the load balancing algorithm from plain Round Robin to Sticky Sessions (using the ip_hash or cookie policies in Caddy):

# Example fallback using sticky cookies if the backend is stateful
example.com {
    reverse_proxy app-1:3000 app-2:3000 {
        # Bind the client to the same backend based on a cookie
        lb_policy cookie {
            name session_binding
        }
    }
}

Special Case Simulation Analysis: Long-lived Connections (WebSockets) #

Round Robin load balancing behavior varies greatly depending on the connection type sent by the client:

• Short HTTP Requests (Rest API): Round Robin works perfectly because each request finishes immediately, so Caddy’s rotation circuit can split the load mathematically 33%-33%-33% very evenly across the three servers.
• Long Persistent Connections (WebSockets / gRPC): Here Round Robin can cause severe load imbalance if one backend is restarted for routine maintenance.

The WebSocket Reconnection Failure Scenario #

Imagine you’re serving 3,000 concurrent active WebSocket connections evenly split by Round Robin to Server A (1,000), Server B (1,000), and Server C (1,000).

1. Server C (:3003) is restarted for an OS update.
2. The 1,000 WebSocket connections on Server C are dropped instantly.
3. Those clients immediately reconnect automatically at the same time.
4. Because Caddy uses plain Round Robin:
   - Reconnection request 1 -> Server A
   - Reconnection request 2 -> Server B
   - Reconnection request 3 -> Server A
   - Reconnection request 4 -> Server B
   (All 1,000 disconnected clients are split evenly to Servers A and B)
5. When Server C comes back up, it's at 0 connections, while Servers A and B carry an extra 1,500 connections each.
6. Round Robin will NOT move WebSocket connections already bound to Server A/B to Server C because WebSocket connections are persistent and never close.

For long-lived persistent connection scenarios like this, switching to the Least Connections (least_conn) algorithm is a far better design decision than keeping Round Robin.

Advanced Health Monitoring Integration #

To keep your Round Robin load balancer from sending traffic to servers experiencing internal damage (e.g., the backend’s database connection is broken but the backend server is still alive), you must create a representative health check endpoint.

Here’s an example of a robust /healthz endpoint implementation using NodeJS/Express for Caddy to poll periodically:

// index.js (NodeJS Backend Application)
const express = require('express');
const mysql = require('mysql2/promise');
const app = express();

// MySQL database connection configuration
const dbPool = mysql.createPool({
    host: process.env.DB_HOST || 'localhost',
    user: 'root',
    database: 'production_db'
});

// Health check endpoint called by Caddy
app.get('/healthz', async (req, res) => {
    const healthInfo = {
        status: 'UP',
        timestamp: new Date().toISOString(),
        details: {
            database: 'UP',
            memory: 'OK'
        }
    };

    try {
        // 1. Test the database connection with a lightweight query
        const connection = await dbPool.getConnection();
        await connection.query('SELECT 1');
        connection.release();
    } catch (err) {
        // If the database is down, mark the health status as DEGRADED
        healthInfo.status = 'DOWN';
        healthInfo.details.database = `ERROR: ${err.message}`;
    }

    // 2. Test the backend server's RAM consumption
    const memoryUsage = process.memoryUsage().heapUsed / 1024 / 1024;
    if (memoryUsage > 500) { // 500MB limit
        healthInfo.status = 'DOWN';
        healthInfo.details.memory = `CRITICAL: Memory usage is ${memoryUsage.toFixed(2)} MB`;
    }

    // Determine the HTTP response status code
    const statusCode = healthInfo.status === 'UP' ? 200 : 503;
    
    // CORRECT: Return a structured response for Caddy validation
    res.status(statusCode).json(healthInfo);
});

app.listen(3000, () => console.log('Backend running on port 3000'));

By returning the 503 Service Unavailable status code when the backend’s internal database has problems, Caddy immediately knows that backend instance isn’t fit to receive normal user traffic and automatically removes it from the Round Robin rotation.

Summary #

• Basic Definition: The Round Robin algorithm routes incoming requests to backend servers alternately in a regular circular rotation.
• Concurrent Performance: Caddy uses low-level atomic operations (sync/atomic) to manage the rotation target index safely without thread locking (lock-free concurrency).
• Load Limitation: Round Robin splits evenly in terms of request count, not computational load weight. Heterogeneous backends should use least_conn instead.
• Architecture Requirement: You must use a stateless architecture (sessions stored in shared Redis/Database) so users aren’t kicked out of their login sessions when moved between backends.
• API Automation: Adding new backends to an autoscaling cluster can be dynamically injected into the rotation via the admin API on port :2019 without a restart.
• Persistent Connections: For long-lived persistent connection protocols like WebSocket and gRPC, Round Robin is prone to load imbalance when a backend returns from a restart.

← Previous: Load Balancer   Next: Least Connections →