ACME Concepts #

The Automated Certificate Management Environment (ACME) protocol is an open standard that defines how a web server proves domain ownership to a Certificate Authority (CA) to obtain TLS certificates automatically. Before ACME, obtaining a TLS certificate was a manual process: error-prone, expensive, and requiring periodic administrator intervention. The standardization of ACME in RFC 8555 transformed the web security landscape by enabling full automation of the certificate lifecycle — from request, verification, and issuance to renewal before expiration.

In the Caddy ecosystem, the ACME protocol is integrated at the system level. Caddy acts as a very smart ACME client, managing the entire process in the background without needing extra tools like cronjobs or external scripts. Understanding the basic concepts of ACME and how its verification works isn’t just academic theory — it’s an essential skill for diagnosing renewal failures, designing secure network architectures, and troubleshooting TLS configuration issues in production.

A Brief History: Before the ACME Era #

To appreciate the efficiency ACME offers, we need to look back at how TLS certificates were obtained and managed in the past. The process wasn’t just time-consuming; it was a significant operational burden for infrastructure teams.

The traditional TLS certificate acquisition process before ACME:

1. Creating the Private Key & CSR:
   The administrator generates a private key and a Certificate Signing Request (CSR) on the server.
   
2. Registration and Payment at a Commercial CA:
   The CSR is sent manually to a commercial CA (like Symantec, DigiCert, Comodo).
   This process requires paying an annual fee ($50 to hundreds of dollars per domain).
   
3. Manual Identity Verification:
   The CA verifies the organization or domain ownership via administrative email,
   phone calls, or legal document submission (which could take days).
   
4. Certificate Installation:
   The issued certificate is sent by email, then downloaded and installed manually
   into the web server configuration (Nginx, Apache, IIS).
   
5. Annual Maintenance and Renewal:
   Certificates usually last 1 to 2 years. Administrators must track expiration dates
   and repeat the entire process above before the validity period ends.

The biggest weakness of this traditional system was the human factor. Many sudden outages (downtime) on major sites happened simply because administrators forgot to renew an expired certificate. The arrival of Let’s Encrypt in 2015 with its ACME protocol succeeded in democratizing HTTPS by making it free, automatic, and accessible to everyone.

The ACME Protocol Architecture (RFC 8555) #

Under the RFC 8555 specification, ACME uses a RESTful architecture over HTTPS. The ACME client (in this case, Caddy) communicates with the ACME Server (like Let’s Encrypt or ZeroSSL) using JSON data formats cryptographically signed with JSON Web Signature (JWS).

The protocol defines several main resource types with structured lifecycles and relationships:

Every Order created by Caddy spawns one or more Authorizations (one for each unique domain name requested). Within each Authorization, the CA offers several Challenge options. Caddy must successfully complete one of those challenges for the Authorization status to change to valid, which then allows certificate issuance.

Core Concept: Domain Validation Mechanisms #

The heart of the ACME protocol is ensuring that the party requesting a certificate truly controls the submitted domain. This proof process is called Domain Validation (DV). The ACME protocol provides three standard challenge methods to verify this ownership.

1. The HTTP-01 Challenge #

The HTTP-01 challenge is the most widely used verification method because of its simplicity. In this method, the CA asks the client to serve a special text document at a specific URL path on the web server in question.

flowchart TD
    A["Request Certificate for example.com"] -->|1. POST /newOrder| B["Provide Token and URL Path"]
    B -->|2. JWS Response| C["Create Token File Automatically"]
    C -->|3. Write Token to URL| D["Perform HTTP GET Verification"]
    D -->|4. HTTP GET to Port 80| C
    D -->|5. Match & Valid| E["Issue TLS Certificate"]
    E -->|6. Send Certificate| A

The detailed mechanism of the flow above is as follows:

1. Caddy sends a certificate request for example.com to the CA.
2. The CA provides a random token and defines the verification path: http://example.com/.well-known/acme-challenge/TOKEN_ID.
3. Caddy automatically intercepts this request internally and prepares a response containing the token signed with the account key (key authorization).
4. The CA makes an HTTP GET connection to port 80 of example.com at that path.
5. If the received token matches the CA’s calculation, domain ownership is declared valid and the certificate is issued.

HTTP-01 Requirements and Limitations:

• Port 80 Access: The standard HTTP port (port 80) on the server must be publicly open and reachable from the global internet.
• DNS Resolution: The domain must already be pointed (have an A/AAAA record) to the public IP of the server running Caddy.
• No Wildcard: This challenge doesn’t support wildcard certificate issuance (e.g., *.example.com) because the CA can’t GET to random hosts dynamically.

2. The TLS-ALPN-01 Challenge #

The TLS-ALPN-01 challenge is designed for situations where HTTP port 80 is blocked by an internet service provider (ISP) or firewall rules, but HTTPS port 443 is freely accessible. This method uses a TLS extension called Application-Layer Protocol Negotiation (ALPN) to negotiate a special protocol named acme-tls/1 during the TLS handshake.

TLS-ALPN-01 Challenge Process Flow:

1. The client (Caddy) creates a certificate request to the CA.
2. The CA responds by providing a special verification token.
3. Caddy creates a temporary self-signed certificate containing that verification token
   in the Subject Alternative Name (SAN) extension with OID 1.3.6.1.5.5.7.1.30.
4. The CA makes a TLS connection to port 443 of the target server.
5. During the TLS handshake, the CA sends an ALPN extension with the value "acme-tls/1".
6. Caddy responds by presenting the temporary self-signed certificate it created.
7. The CA verifies the signature and token inside that certificate. If valid,
   the real certificate is issued and Caddy removes the temporary certificate from memory.

TLS-ALPN-01 Requirements and Limitations:

• Port 443 Access: The standard HTTPS port (port 443) must be open and directly reachable from the internet.
• No Layer 7 Proxy: The TLS connection must be terminated directly by Caddy. If a reverse proxy or CDN (like Cloudflare with flexible encryption/TLS termination) sits in front of Caddy, this challenge fails because the CA negotiates with the proxy, not with Caddy.
• No Wildcard: Same as HTTP-01, this method can’t be used to verify wildcard certificates.

3. The DNS-01 Challenge #

The DNS-01 challenge proves domain ownership by creating a special text record (TXT record) in the domain’s DNS zone. This is the most flexible and powerful method of the three.

flowchart TD
    A["Request Certificate for *.example.com"] -->|1. Order| B["Provide Challenge Token"]
    B -->|2. Token| C["Create TXT Record via DNS API"]
    C -->|3. API Call| D["Publish TXT Record"]
    D -.->|4. DNS Propagation| E["Perform DNS TXT Query"]
    E -->|5. Query TXT| D
    E -->|6. Valid| F["Issue TLS Certificate"]
    F -->|7. Certificate| A
    A -->|8. Done| G["Remove TXT Record (Cleanup)"]

The DNS-01 execution steps:

1. Caddy requests a certificate for the domain *.example.com (wildcard) or a standard domain.
2. The CA sends a cryptographic challenge token.
3. Caddy uses the configured DNS API plugin to create a new TXT record named _acme-challenge.example.com containing that token value.
4. Caddy waits a while for the DNS record to propagate.
5. The CA queries the DNS TXT record at the domain’s nameservers.
6. If the TXT record value matches, verification succeeds and the certificate is issued.
7. Caddy removes the TXT record again to keep the DNS zone clean.

DNS-01 Advantages:

• The Only Option for Wildcards: This is the only method approved by CAs for wildcard certificate issuance (*.domain.com).
• Hidden Servers (Private Networks): The Caddy server doesn’t need open inbound ports 80 or 443 from the internet. Servers on local networks (LAN) or behind strict firewalls can still get valid public certificates as long as they have outbound access to call the DNS API and the ACME API.

Multi-Perspective Validation (MPV) #

One crucial security aspect adopted by Let’s Encrypt and other modern CAs is Multi-Perspective Validation (MPV), sometimes called Multi-Perspective Verification.

Why Is MPV Needed? #

In the early days of ACME, challenge verification (e.g., HTTP-01) was only performed from a single CA data center location. This single-point structure had a serious security gap against advanced network attacks like BGP Hijacking or localized DNS Spoofing.

If an attacker manages to hijack the BGP route for your server’s IP near the CA’s data center, they can redirect verification traffic to their own server, answer the ACME challenge, and obtain a legitimate TLS certificate for your domain without permission.

How MPV Works #

To mitigate this risk, the CA performs challenge verification from several different geographic regions simultaneously:

flowchart TD
    Primary["Let's Encrypt US-East (Primary)"]
    
    subgraph Satellites["Satellites Nodes"]
        direction LR
        West["Let's Encrypt US-West"]
        Europe["Let's Encrypt Europe"]
        Asia["Let's Encrypt Asia"]
    end

    Primary -->|"Trigger verification"| West
    Primary -->|"Trigger verification"| Europe
    Primary -->|"Trigger verification"| Asia
    
    West -. "GET / DNS Query" .-> Caddy["Our Caddy Server"]
    Europe -. "GET / DNS Query" .-> Caddy
    Asia -. "GET / DNS Query" .-> Caddy

    style Primary stroke:#0288d1,stroke-width:2px
    style Caddy stroke:#43a047,stroke-width:2px
    style Satellites stroke:#757575,stroke-width:1px,stroke-dasharray:5,5

The main CA (for example, on the US East Coast) asks several of its satellite nodes worldwide (for example, US West Coast, Europe, and Asia) to independently perform challenge verification. Certificates are only issued if all or a majority of those nodes report successful, consistent verification results.

[!WARNING] MPV makes the practice of whitelisting Let’s Encrypt IPs on your firewall an anti-pattern. Because Let’s Encrypt uses many dynamically changing satellite IPs to validate from various global vantage points, any attempt to restrict verification access to specific IPs will inevitably cause renewal failures. The correct approach is to keep port 80 open to all public IPs, or switch to the DNS-01 challenge, which needs no open ports.

The Certificate Lifecycle in Caddy #

Caddy’s certificate lifecycle automation is designed to minimize human involvement. The image below explains how Caddy makes decisions at first startup or when serving a new domain request.

flowchart TD
    A["Start Caddy / Detect New Configuration"] --> B{"Does the certificate exist in local storage?"}
    B -- Yes --> C{"Is the remaining validity > 30 days?"}
    B -- No --> E["Create ACME Account & Place Order"]
    C -- Yes --> D["Use Certificate for TLS Handshake"]
    C -- No --> F["Trigger Background Renewal"]
    E --> G["Choose CA (Let's Encrypt / Fallback ZeroSSL)"]
    G --> H["Select and Run Challenge (HTTP/TLS/DNS)"]
    H --> I{"Did verification succeed?"}
    I -- Yes --> J["Save New Certificate to Storage"]
    I -- No --> K{"Is there an Alternative CA?"}
    K -- Yes --> L["Switch to Alternative CA"]
    L --> H
    K -- No --> M["Log Error & Schedule Retry with Backoff"]
    J --> D
    F --> E

Detailed Lifecycle Explanation: #

1. Local Storage Check: Caddy always checks its storage folder before making any outbound connection. Caddy’s default storage is on the local filesystem, securely isolated from regular user access.
2. Renewal Trigger (Renewal Window): The validity period of ACME certificates (Let’s Encrypt and ZeroSSL) is 90 days. Caddy proactively starts the renewal process when a certificate enters its last 30 days of validity (after 60 days of active life). This wide window provides enough tolerance for potential network disruptions or temporary issues on the CA side.
3. Exponential Backoff Retry Mechanism: If certificate acquisition fails (for example, DNS hasn’t propagated or the CA server is down), Caddy doesn’t keep flooding the CA with new requests. Caddy reschedules the next attempt using an exponential backoff algorithm: first waiting 1 minute, then 2 minutes, 4 minutes, up to a maximum interval of a few hours.

Let’s Encrypt Rate Limit Rules and Boundaries #

To protect their infrastructure from abuse and denial-of-service attacks, Let’s Encrypt enforces strict rate limits on its production environment. Understanding these boundaries is essential so you don’t lock yourself out of HTTPS access during migrations or system testing.

Let’s Encrypt Rate Limit Table (Production) #

Anti-Pattern in Container Environments (Docker) #

One of the most common causes of exhausting Let’s Encrypt rate limits is negligence in managing data volumes in container environments like Docker or Kubernetes.

# ANTI-PATTERN: Running a Caddy container without data volume persistence
services:
  caddy:
    image: caddy:2.8.4
    ports:
      - "80:80"
      - "443:443"
    # ✗ DON'T: No volume for Caddy's /data directory

If the container above is stopped and recreated (for example, every time you update the application), all certificates already obtained are lost from the container. When the new Caddy starts, it’s forced to request new certificates from Let’s Encrypt. After 5 container restart cycles in a week, you hit the Duplicate Certificate limit and your site can’t be accessed over HTTPS for an entire week.

# CORRECT: Always use named volumes for Caddy's data and config folders
services:
  caddy:
    image: caddy:2.8.4
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data          # ✓ STORES CERTIFICATES AND ACCOUNT KEYS
      - caddy_config:/config      # ✓ STORES ACTIVE CONFIGURATION

volumes:
  caddy_data:
  caddy_config:

Using the Staging Environment for Testing #

When you’re testing deployment scripts, new Caddyfile configurations, or large server migrations, always use Let’s Encrypt’s Staging Environment. The staging environment has much larger rate limits (up to 30,000 certificates per domain per week), so it’s safe from lockout risk.

You can enable the staging environment by specifying the staging directory URL in the Caddyfile global options:

{
    # Use the Let's Encrypt staging server for testing
    acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
    email [email protected]
}

test.example.com {
    file_server
}

[!WARNING] Certificates issued from the staging environment are not trusted by browsers. Browsers show a red security warning (Untrusted Connection). Use this environment purely to confirm that the ACME verification flow (HTTP/TLS/DNS) runs successfully without errors in the Caddy log. Once proven successful, remove the acme_ca configuration and reload Caddy to get the official (production) certificate.

ACME Troubleshooting and Debugging #

When Caddy fails to obtain a certificate, the first step is analyzing the system logs. Caddy provides very descriptive error messages showing where in the ACME process it stopped.

Essential Diagnostic Commands #

Here are some terminal commands you must run to track down ACME problems:

# 1. Check Caddy logs in real time to detect verification failures
sudo journalctl -u caddy -f --no-pager | grep -i "acme\|certificate\|challenge"

# 2. For Docker-based deployments, check the container logs
docker logs caddy 2>&1 | grep -i -E "error|warn|acme"

# 3. Check whether the public domain already points to your server IP
dig +short example.com
nslookup example.com

# 4. Verify port 80 is reachable from the outside internet
# (Run this command from an external computer or another server)
curl -Iv http://example.com/.well-known/acme-challenge/test-check

# 5. Check other applications that might be monopolizing ports 80 or 443
sudo ss -tlnp | grep -E ':80|:443'

Common ACME Error Classification #

When analyzing logs, you’ll often encounter several standard errors from the ACME server. Here are the meanings and solutions for each:

1. urn:ietf:params:acme:error:connection #

• Cause: The CA server can’t reach your Caddy server on port 80 (HTTP-01) or port 443 (TLS-ALPN-01).
• Solution: Check your server’s firewall rules (e.g., ufw, iptables, AWS Security Group, or Cloudflare SSL settings). Make sure traffic from any internet IP is allowed into those ports.

2. urn:ietf:params:acme:error:dns #

• Cause: The CA server failed to resolve DNS for the requested domain, or couldn’t find the expected TXT record when using the DNS-01 challenge.
• Solution: Double-check the domain name in the Caddyfile. If you just changed DNS records, wait for full propagation or clear your local resolver’s DNS cache.

3. urn:ietf:params:acme:error:rateLimited #

• Cause: You’ve exceeded one of the frequency limits set by Let’s Encrypt.
• Solution: Temporarily switch the configuration to the staging environment or use ZeroSSL as an alternative CA until the reset period passes (usually 7 days for the registered domain limit).

Summary #

• The ACME Protocol (RFC 8555) — The open standard used by Let’s Encrypt and ZeroSSL to fully automate the TLS certificate lifecycle.
• The HTTP-01 Challenge — The domain proof method using a token file on port 80. This is Caddy’s built-in (default) method. Doesn’t support wildcards.
• The TLS-ALPN-01 Challenge — The proof method using TLS handshake protocol negotiation on port 443. Useful when port 80 is blocked. Doesn’t support wildcards and must not be behind a Layer 7 proxy.
• The DNS-01 Challenge — The proof method using DNS TXT records. Mandatory for wildcard certificates (*.domain.com) and servers on private networks.
• Multi-Perspective Validation (MPV) — The CA technique of validating challenges from several geographic regions to prevent network hijacking attacks (BGP hijacking). Don’t restrict CA IPs on your firewall.
• Lifecycle Management — Caddy intelligently detects certificate status, starts automatic renewal 30 days before validity ends, and retries regularly if failures occur.
• Persistent Storage — Always point the /data volume in Docker containers to persistent host storage so certificates aren’t wasted and rate limit blocks are avoided.

← Previous: Automatic HTTPS   Next: Let’s Encrypt →