Automating Certificates at Scale with ACME

Consider the arithmetic, because the arithmetic, more than the cryptography, decides whether a certificate estate is manageable. Take a modest environment: two thousand TLS endpoints. Now apply a ninety-day certificate lifetime. That is roughly twenty-two renewals every single day, seven days a week, before anyone has poured a coffee. Halve the lifetime, as the industry is steadily doing, and that figure doubles.

No team renews twenty-plus certificates a day by hand. Most cannot even renew two a day reliably; it takes only a single high-profile expiry slipping through to take down a payment flow for an afternoon. So automation isn't an optimisation. At short validity and real scale, manual certificate management does not degrade gracefully. It collapses.

This is the part of the series where the focus shifts from describing the lifecycle to operating it. The protocol that makes that possible is ACME, and most of this article is about how to build a platform around it rather than how to run a single client.

ACME, precisely: RFC 8555 and its object model

ACME (the Automatic Certificate Management Environment, standardised as RFC 8555) is at heart a small REST-ish protocol for proving you control an identifier and getting a certificate for it without a human in the loop. Every request from the client is a JWS (JSON Web Signature) object, signed either by the account key or, for the initial registration, by the key embedded in the request. That signing is what gives the CA a verifiable, replay-protected record of who asked for what.

The object model is worth learning properly, because the whole protocol is transitions between these resources:

• Account: keyed by an asymmetric account key, established once. Everything else is created under it.
• Order: a request for a certificate covering one or more identifiers (DNS names, and increasingly IP addresses). An order moves through pending → ready → processing → valid.
• Authorization: one per identifier in the order, representing the CA's question "do you actually control this name?"
• Challenge: the concrete way you answer that question for an authorization: HTTP-01, DNS-01 or TLS-ALPN-01.
• Certificate: the issued chain, downloadable once the order is valid and finalised.

Proving control: the three challenge types

All three challenges prove the same thing (that you control the identifier), but they make very different operational trade-offs, and choosing badly is one of the most common automation mistakes.

• HTTP-01: the client serves a key authorization at /.well-known/acme-challenge/<token> over port 80. Simple, but it requires inbound port 80 and cannot issue wildcards.
• DNS-01: the client publishes a TXT record at _acme-challenge.<domain> whose value is a hash of the key authorization. It is the only challenge that issues wildcards, and the only one that works for endpoints with no inbound reachability, which describes most of an internal estate. The price is that your automation needs programmatic control of DNS.
• TLS-ALPN-01: the client answers on port 443 using a special ALPN protocol identifier and a self-signed certificate carrying the validation value. Useful when 443 is the only port you can rely on and you want to keep validation entirely in the TLS layer.

For any sizeable internal estate, my default is DNS-01. It decouples validation from the endpoint's network position entirely, which is exactly what you want when the thing needing a certificate is a database replica three subnets deep with no path from the public internet.

The issuance flow end to end

Strip away the JSON and the flow is the same every time:

  ACME Client                          ACME Server (CA)
      |                                       |
      |  1. newNonce (HEAD)                   |
      |-------------------------------------->|
      |  2. newAccount (JWS, account key)     |
      |-------------------------------------->|
      |<-------------------------------------- account URL
      |                                       |
      |  3. newOrder  [example.com, *.x.com]  |
      |-------------------------------------->|
      |<-------------------------------------- order: status=pending
      |                                       |  + authorization URLs
      |                                       |
      |  4. GET authorization                 |
      |-------------------------------------->|
      |<-------------------------------------- challenges (http/dns/tls-alpn)
      |                                       |
      |  5. provision challenge               |
      |     (TXT record / .well-known file)   |
      |                                       |
      |  6. POST challenge "I am ready"       |
      |-------------------------------------->|
      |                                       |--- CA validates control
      |<-------------------------------------- authorization: status=valid
      |                                       |
      |  7. finalize (POST CSR)               |
      |-------------------------------------->|
      |<-------------------------------------- order: status=valid + cert URL
      |                                       |
      |  8. GET certificate (download chain)  |
      |-------------------------------------->|
      |<-------------------------------------- PEM chain
      |                                       |

The ordering matters: the CSR is submitted at finalize, not at order creation. The order names the identifiers, and the CSR (and with it the key pair) only appears once control has been proven. That separation matters when you design key handling, because it means the private key never has to leave the host that will use it.

Architecting a certificate-automation platform

A single ACME client on a single host is a solved problem you can stand up in an afternoon. A platform that keeps thousands of certificates valid across a heterogeneous estate is not, and it is where the real engineering lives. I find it easiest to think of as six cooperating concerns.

Discovery and inventory

This comes first, and it is the part most teams skip. You cannot automate what you cannot see. Before any of the clever renewal logic earns its keep, you need a continuously updated inventory of every certificate in the estate: where it lives, what it covers, who issued it, when it expires, and, most importantly, whether anything is automating it. Build this from active network scanning, CT log monitoring for your own domains, and querying the issuance service's own records, then reconcile the three. The certificates that hurt you are the ones not in any of those lists.

The certificate that takes an estate down at 3am is never the one on the dashboard. It is the one nobody knew existed, last renewed by a person who left eighteen months ago.

The issuance service and lightweight agents

Centralise policy, distribute execution. The issuance service is the control plane: it holds ACME account credentials, enforces naming and key-type policy, brokers DNS-01 provisioning through a privileged path, and records every issuance into inventory. On each host sits a lightweight agent whose job is deliberately small: request, install, reload, report. Keep the agent dumb and the service opinionated. An agent that makes policy decisions is an agent you will be debugging across ten thousand hosts at once.

The renewal scheduler

Renew early. A widely used convention is to renew at roughly one third of the certificate's remaining lifetime. For a ninety-day certificate, that means attempting renewal around the thirty-day-remaining mark, leaving a thirty-day window in which to retry before anything actually expires. That window is your recovery budget.

Just as important: add jitter. If every agent renews at exactly thirty days, you build a thundering herd that hammers the CA and trips its rate limits in synchronised waves. Spread renewals across the window randomly.

# Example renewal-decision logic (pseudocode)
lifetime      = not_after - not_before
renew_after   = not_before + (lifetime / 3) * 2   # ~1/3 life remaining
jitter        = random_minutes(0, 720)            # up to 12h spread

if now >= (renew_after + jitter):
    try:
        new_cert = acme.renew(order)
        install(new_cert)
        reload_service()           # SIGHUP / socket reload, not full restart
        report_success(inventory)
    except AcmeError as e:
        backoff_and_retry(e)       # exponential, respect Retry-After
        report_failure(alerting)   # alert NOW, with 30 days still in hand

Key generation, storage and deployment hooks

Generate the key pair on the host that will use it, wherever you can. The private key should not travel; if it never leaves the box, you have one fewer thing to protect in transit and one fewer copy to revoke when something leaks. Where hardware or a secrets manager is in play, generate inside that boundary and hand back only a handle. After installation, the deployment hook should reload rather than restart the consuming service. A graceful reload keeps connections alive, whereas a restart is a small self-inflicted outage repeated thousands of times a day.

Monitoring and alerting

The most important inversion in this article is to alert on renewal failure, not on expiry. Expiry monitoring tells you about a fire once the building is already warm. If you renew at one third of lifetime and alert the moment a renewal attempt fails, you learn about the problem with weeks of runway to fix it calmly. An expiry alert at seven days is a panic; a renewal-failure alert at thirty days is a ticket. Keep the expiry monitor too, but only as the backstop that catches the things your automation never knew about.

ACME behind the firewall: private and internal PKI

ACME is too often filed under "the public web", the thing that gets you a free certificate for a public site. That framing misses the larger prize. ACME is just a protocol, and an internal CA can speak it exactly as a public one does. Point your agents' ACME directory URL at the internal CA instead of a public endpoint and the entire automation machine works identically behind the firewall. Same orders, same DNS-01 challenges, same renewal scheduler, but now issuing the short-lived internal certificates that secure service-to-service traffic, mutual TLS, message brokers and the rest of the private estate.

The internal case is in some ways the stronger one. You are not bound by public CA/Browser Forum constraints, so you can choose aggressively short lifetimes, days or even hours, because automation makes the renewal cost effectively zero. Short-lived internal certificates are one of the cleanest ways to reduce your dependence on revocation infrastructure, which has historically been difficult to scale and rely on.

The one genuinely new problem internal ACME creates is trust distribution. A public CA is already in every trust store; your internal root is in none of them. So the internal root and issuing intermediates must be pushed to every consuming trust store (OS stores, language runtimes, container base images) through configuration management, and rotated there before the CA itself rotates. Treat the trust bundle as a managed, versioned artefact in its own right, not a file someone copied to a host in 2019.

Failing safely: retries, backoff and rate limits

Test automation under failure, not just on the happy path. ACME issuance touches DNS, network paths, a CA that may be busy, and your own provisioning code, and all of them fail intermittently. The platform has to assume failure is routine.

• Retry with exponential backoff, and honour Retry-After. When the CA returns a 429 or a Retry-After header, that is not advisory. Hammering through it is how you turn a transient slowdown into a sustained rate-limit ban.
• Respect rate limits as a design input. Public ACME endpoints cap orders per account and per registered domain over a sliding window; internal CAs have their own ceilings. Your scheduler's jitter and your backoff are what keep aggregate request rates under those caps.
• Handle partial failure explicitly. A multi-identifier order where one authorization fails should not silently ship a certificate missing a name, nor should it discard the authorizations that did succeed. Fail the order cleanly, keep what is reusable, and surface exactly which identifier could not be validated.
• Degrade with lead time. This is the payoff of renewing early. A failed renewal at one third of lifetime is a problem you address during working hours; the same failure with no lead time is an outage. The lead time is the safety mechanism, so design every retry and alert to consume it slowly and visibly.

Anti-patterns that keep recurring

These turn up in estate after estate, and each one is a slow-motion incident waiting for its date:

• Cron-and-pray. A renewal job in cron with its output redirected to /dev/null and no alerting. It works for two years, fails silently in the third, and you find out from a customer.
• Snowflake renewals. Every team automating differently, with bespoke scripts and no shared inventory or policy. Ten automation systems are harder to trust than one.
• No inventory. Automating issuance while having no authoritative view of what exists. The certificates outside your system are precisely the ones that will expire unwatched.
• Keys copied around. One private key generated centrally and copied to fifty hosts. Now a single compromise is fifty revocations, and you have lost track of which hosts even hold it.
• Restart-on-renewal. Bouncing a service on every renewal instead of a graceful reload, converting routine automation into routine micro-outages.

Done well, certificate automation becomes the quiet layer nobody thinks about: certificates appear, renew and rotate without a ticket, and the estate stays valid. But getting there exposes a deeper requirement: an automated estate can only rotate as fast as its slowest assumption about keys, algorithms and trust. Once renewal is cheap and continuous, the question stops being "can we renew in time?" and becomes "can we change what we issue (the algorithms, the key types, the chains) just as fluidly?" That is crypto-agility, and it is where this series turns next.