IP Hash #

Keeping users always connected to the same backend server during their interaction session is a crucial need for traditional web applications storing session state in local server memory. Caddy provides the IP Hash algorithm (ip_hash) as a session-affinity-based (session stickiness) load balancing solution. We’ll learn about the mathematical formula behind IP hash mapping, the hidden danger of NAT network gateways (NAT Gateway Bottleneck), secure configuration using trusted_proxies, and cookie-based sticky session alternatives for modern architectures.

How the IP Hash Algorithm Works #

Conceptually, IP Hash works by converting the client’s dynamic IP address into a static numeric index value pointing to one of the backend (upstream) servers.

1. FNV-1a Hashing and Modulo Calculation #

In Caddy’s Go code, this hashing process uses the FNV-1a (Fowler-Noll-Vo) algorithm, which produces a 32-bit hash that’s very evenly distributed with fast computation. The algorithm starts with the offset basis value 2166136261 and multiplies by the FNV prime constant 16777619 at each byte processing step.

Why FNV-1a Instead of SHA-256? #

When designing high-performance network gateways, the hash function choice is crucial. Cryptographic algorithms like MD5, SHA-1, or SHA-256 are designed to have high collision resistance and encryption security, but they demand massive CPU processing power. In contrast, FNV-1a is a non-cryptographic hash function. FNV-1a is designed to run in just a few low-level CPU instructions, making it very fast and efficient on high-traffic servers for mapping network addresses without burning CPU cycles.

Here’s a step-by-step FNV-1a calculation table for the client IP address 203.0.113.5:

After the last byte is processed, we get the integer hash value 394208453. To determine the target backend server from 3 registered backends, Caddy uses the modulo formula:

$$\text{Upstream Index} = 394208453 \pmod 3 = 2$$

The modulo result 2 points to the third backend (index 2). The Python script below can be used to simulate this calculation locally:

# fnv_calculator.py
def calculate_fnv1a_32(ip_string):
    # Convert the IP string into a byte array
    ip_bytes = [int(b) for b in ip_string.split('.')]
    
    # FNV-1a initialization values
    h = 2166136261
    prime = 16777619
    
    for byte in ip_bytes:
        h = h ^ byte
        h = (h * prime) & 0xffffffff  # Limit to 32-bit unsigned
        
    return h

ip = "203.0.113.5"
hash_val = calculate_fnv1a_32(ip)
num_backends = 3
backend_index = hash_val % num_backends

print(f"IP: {ip} -> FNV-1a Hash: {hash_val} -> Modulo {num_backends} = Index: {backend_index}")

2. The IP Roaming Issue on Mobile Networks #

Although FNV-1a is very fast, its dependence on the IP address creates an issue for mobile device users (smartphones). When users ride a train or public transport, their smartphone keeps moving from one cell tower (Base Transceiver Station - BTS) to the next. This handover process forces the mobile operator to dynamically assign new public IP addresses to clients:

• IP at Location A: 103.20.10.15 -> FNV Hash: 39281045 -> Modulo 3 = Index 2 (Server 3)
• IP at Location B: 103.20.10.22 -> FNV Hash: 29841022 -> Modulo 3 = Index 0 (Server 1)

Due to this dynamic IP change, mobile users get kicked out of their login sessions repeatedly because Caddy keeps moving their routes to different backends.

3. IPv6 Masking and Client Privacy (RFC 4941) #

Modern operating systems use IPv6 Privacy Extensions (RFC 4941), which randomly change a client device’s IPv6 address every few hours to protect user privacy on the internet. If Caddy hashes the full IPv6 address (/128), client connections get disconnected from the backend server they were bound to every time the OS changes its IP address.

To solve this issue, Caddy intelligently masks the IPv6 address by only taking the first 64 bits (/64 subnet prefix) to hash:

flowchart TD
    ClientIP["Client's Original IPv6 Address: 2001:db8:85a3:8d3:1319:8a2e:370:7348"] -->|"Caddy truncates to /64"| HashIP["Hashed Address: 2001:db8:85a3:8d3::"]

Because the first 64 bits represent the client’s stable home/office network address, users stay connected to the same backend even when their device’s internet identification token changes dynamically.

flowchart TD
    Client1["Client A: 203.0.113.5"] --> Caddy{"Caddy Proxy"}
    Client2["Client B: 198.51.100.12"] --> Caddy

    subgraph Hashing_Engine["Caddy IP Hashing Engine"]
        H1["Hash(203.0.113.5) % 3"] -->|Index 0| U1["Server 1"]
        H2["Hash(198.51.100.12) % 3"] -->|Index 2| U3["Server 3"]
        U2["Server 2 (Index 1)"]
    end

    Caddy --> H1
    Caddy --> H2

    style U1 stroke:#0288d1,stroke-width:2px
    style U2 stroke:#0288d1,stroke-width:2px
    style U3 stroke:#0288d1,stroke-width:2px

IP Hash Configuration #

To enable the IP hash-based load balancing policy, just set the lb_policy ip_hash parameter on the reverse_proxy directive in your Caddyfile:

# Reverse proxy configuration with IP Hash
example.com {
    reverse_proxy backend-1:3000 backend-2:3000 backend-3:3000 {
        # Enable IP Hash load balancing
        lb_policy ip_hash
        
        # Active health monitoring configuration (highly recommended)
        health_uri      /healthz
        health_interval 10s
        health_timeout  3s
    }
}

Adding an active health check is crucial when using ip_hash. If one backend dies undetected, Caddy keeps trying to send (hashed) bound clients’ requests to the dead server, causing persistent 502 Bad Gateway errors for those users. With health checks, Caddy immediately removes the dead backend from the modulo calculation circuit and automatically re-hashes clients to other healthy backends.

The Important Role of trusted_proxies #

The ip_hash configuration will fail completely in production if you place Caddy behind an external proxy service like the Cloudflare CDN, an AWS Application Load Balancer (ALB), or an Nginx gateway without defining the trusted_proxies block.

Right-to-Left Client IP Scanning #

When passing through several proxies, the X-Forwarded-For header contains a comma-separated list of IPs. To safely detect the real client IP without IP manipulation, Caddy applies a right-to-left scanning algorithm:

X-Forwarded-For Header Contents: 203.0.113.5, 172.16.0.22, 10.0.0.10
                                                        ▲
                                              (Caddy starts scanning)

1. Caddy reads the rightmost IP (10.0.0.10).
2. Caddy checks: Is 10.0.0.10 on the trusted_proxies list?
3. If YES (trusted), Caddy shifts left to read 172.16.0.22.
4. Caddy checks: Is 172.16.0.22 trusted?
5. If NO (untrusted), Caddy stops scanning and designates 172.16.0.22 as the legitimate real client IP to hash.

Register the proxy IP ranges in front of Caddy in the global options so Caddy extracts the real IP from the X-Forwarded-For header before computing the hash value:

# Caddyfile global options block
{
    servers {
        # Example of trusting the local load balancer IP (10.0.0.0/24)
        trusted_proxies static 10.0.0.0/24
    }
}

# Site block
example.com {
    reverse_proxy backend-1:3000 backend-2:3000 {
        lb_policy ip_hash
    }
}

The NAT Gateway Bottleneck Problem #

Even with trusted_proxies correctly configured, the ip_hash algorithm architecturally has an inherent weakness that can’t be avoided when facing NAT (Network Address Translation) Gateways.

In the real world, thousands of physical users in one large office building, university, or mobile network (like 4G/5G carriers) access the public internet using the same public IP address installed on their main NAT gateway.

flowchart TD
    subgraph Big_Office["Office Network (Behind NAT)"]
        U1["User 1 (Local IP: 192.168.1.10)"]
        U2["User 2 (Local IP: 192.168.1.11)"]
        U3["User 3 (Local IP: 192.168.1.12)"]
        
        NAT["Office NAT Gateway\n(Public IP: 103.10.20.30)"]
        
        U1 --> NAT
        U2 --> NAT
        U3 --> NAT
    end

    NAT -->|"All requests have IP 103.10.20.30"| Caddy{"Caddy Proxy"}

    subgraph Backend_Cluster["Backend Cluster"]
        B1["Server 1 (Overwhelmed!)"]
        B2["Server 2 (Idle)"]
        B3["Server 3 (Idle)"]
    end

    Caddy -->|"Hash(103.10.20.30) % 3 -> Server 1"| B1
    Caddy -.-> B2
    Caddy -.-> B3

    style B1 stroke:#e53935,stroke-width:2px

Because Caddy receives all requests from thousands of office employees labeled with the same originating public IP (103.10.20.30), Caddy computes an identical hash for all of them. The result: every employee in that building gets locked onto the same backend server, creating an extreme workload hotspot (load imbalance).

The Best Alternative: Sticky Cookie Sessions #

To avoid the load imbalance problem caused by NAT Gateways, modern application architectures prefer Cookie-based Sticky Sessions over IP Hashing.

With the cookie policy, Caddy doesn’t care about the client’s IP address. On the first request, Caddy generates a unique encrypted tracking cookie, inserts it into the client’s browser, and uses it to recognize the client session on subsequent requests.

You can secure that cookie by applying strict browser security attributes like HttpOnly (prevents cookie theft via JavaScript XSS), Secure (only sent via HTTPS), and SameSite=Strict (CSRF protection):

# Securing sticky sessions with a cookie heavy on security attributes
example.com {
    reverse_proxy backend-1:3000 backend-2:3000 {
        # Set the load balancing policy to cookie
        lb_policy cookie {
            name     caddy_session
            secret   "production-secret-key-must-be-long-and-unique-once"
            
            # Cookie expiration time (e.g., 1 hour)
            # cookie_lifetime 3600
        }
        
        # Inject additional security attributes into the cookie via custom header_down if needed
        # header_down Set-Cookie "caddy_session; Path=/; Secure; HttpOnly; SameSite=Strict"
    }
}

Cookie Handling on the Backend Side #

Here’s an example of how your Node.js backend server can detect whether a request is bound to the correct session using the cookie header forwarded by Caddy:

// NodeJS Backend Server
const express = require('express');
const cookieParser = require('cookie-parser');
const app = express();

app.use(cookieParser());

app.get('/api/session', (req, res) => {
    // Read the sticky session pointer cookie set by Caddy
    const caddySession = req.cookies['caddy_session'];
    
    console.log(`[Backend-1] Received request with Caddy Session ID: ${caddySession}`);
    
    res.json({
        message: "Request successfully processed at Backend-1",
        session_id: caddySession
    });
});

app.listen(3000);

Comparison Table: ip_hash vs cookie #

Production Failure Case Study Due to Wrong IP Hash Configuration #

To better understand the importance of these architectural details, here’s a post-mortem analysis chronology of a real system failure:

[ POST-MORTEM EVENT LOG: INCIDENT #10892 ]
Topic: API Cluster Load Failure (Overload) On Server 1
System: Caddy Proxy v2.7.x behind Cloudflare CDN

1. PROBLEM SYMPTOMS:
   - At 9:00 AM, the Prometheus monitoring alarm went off.
   - Backend Server 1 hit 100% CPU utilization and full memory (OOM).
   - Backend Server 2 and Backend Server 3 recorded less than 2% CPU usage.
   - Clients received slow responses with 504 Gateway Timeout status.

2. INVESTIGATION & ROOT CAUSE:
   - DevOps checked the Caddyfile configuration and found:
     reverse_proxy app1:3000 app2:3000 app3:3000 {
         lb_policy ip_hash
     }
   - However, the global options didn't define "trusted_proxies".
   - As a result, Caddy ignored the "X-Forwarded-For" header sent by Cloudflare.
   - Caddy computed the FNV-1a Hash from the direct TCP connection source IP, which was the Cloudflare Edge server IP.
   - Because all requests from thousands of clients came through Cloudflare, the source IP in Caddy's eyes was the same.
   - The modulo calculation result locked all user requests to Backend Server 1.

3. IMMEDIATE REMEDIATION:
   - Added the "trusted_proxies static <Cloudflare IP>" block to the Caddyfile global options.
   - The Caddyfile was dynamically reloaded without a restart via the Admin API.
   - The workload evened out instantly within 2 seconds across all Backends.

4. LESSONS LEARNED:
   - Must periodically sync the Cloudflare IP list using an automated orchestration script.
   - For web clusters with dominant mobile users, migrate the "ip_hash" configuration to "cookie" to prevent mobile IP distribution bias.
   - Add a Load Skew Indicator to the Prometheus monitoring dashboard with the metric:
     caddy_reverse_proxy_upstreams_requests_active
   - Create an automatic alarm if the active request load difference between backends exceeds 30% within a 5-minute window.

Handling Downstream Upstreams (Failover) #

One of the biggest concerns with sticky sessions is how Caddy handles emergency situations when the backend server a client is locked to suddenly crashes (failover).

Caddy handles this scenario very gracefully without disrupting user service:

1. If Server A (where Client A is bound) dies, Caddy detects the failure through the health check module.
2. Caddy removes Server A from the modulo hashing calculation list.
3. When Client A sends the next request, Caddy detects that the previous destination backend isn’t available.
4. Caddy performs a re-hash using the remaining healthy backend list (Server B and Server C).
5. Client A is transparently shifted to Server B.
6. Important Note: Because Client A’s session was stored in the local memory of the dead Server A, Client A must log in again on Server B (unless your application uses a Redis shared session store). However, at least the web service remains accessible and doesn’t show an error page.

Summary #

• Sticky Sessions: The IP Hash algorithm consistently maps client IPs to the same backend server using mathematical modulo calculations.
• Hidden NAT Danger: Thousands of users behind one office/mobile NAT gateway share the same public IP, triggering workload stacking on a single server.
• trusted_proxies Role: Must be configured when behind a CDN/Load Balancer so Caddy doesn’t hash the load balancer’s IP address.
• IPv6 Masking: Caddy automatically truncates the subnet mask to the /64 range to stabilize IP hashing on mobile device connections.
• Cookie Alternative: The lb_policy cookie policy is the best solution for overcoming load imbalance caused by NAT Bottlenecks in production.
• Graceful Failover: If the target backend server dies, Caddy re-computes the hash to transparently shift clients to other healthy backends.

← Previous: Least Connections   Next: Weighted →