Across this series we have spent five articles building things: anatomising a modern PKI, walking the certificate lifecycle, automating issuance with ACME, designing for crypto-agility, and extending identity to machines and IoT fleets. This final part is about the part nobody puts on the architecture diagram and everybody underestimates: what happens after go-live. Cutting over to your new hierarchy feels like the finish line, but it is really the moment your PKI stops being a project and becomes an operational discipline you will run for a decade.
Well-designed PKIs can rot within a year or two when nobody owns them, while unglamorous ones run reliably for years when the operating model is treated as seriously as the cryptography. The maths is rarely what separates them. What does is governance, ceremony discipline, observability, and a rehearsed answer to the question every architect should be able to answer: what do we do when a key we trusted is no longer trustworthy?
So this is the part where we stop drawing boxes and start talking about running them.
Governance: policy is the contract for trust
Every trustworthy PKI rests on two documents, and many in-house PKIs have neither written down properly. The Certificate Policy (CP) states what your certificates mean, i.e. the rules a certificate must satisfy to be considered fit for a given purpose. The Certification Practice Statement (CPS) states how you actually operate the CA to meet that policy: identity vetting, key generation, validity periods, revocation handling, custody of key material. The CP is the promise; the CPS is the evidence that you keep it.
This matters because trust is transitive. When a relying party validates a chain, it is implicitly accepting your CP. If the policy is vague or unwritten, you have no defensible answer when someone asks "why should I trust a certificate from this issuer for this use?" Writing the CP forces decisions you would otherwise discover during an incident.
Designing the hierarchy deliberately
The hierarchy is policy made structural. Do not let it emerge by accident. A few principles worth holding firmly:
- One offline root, kept offline. It signs intermediates and nothing else. Its job is to be boring and rarely used.
- Intermediates per purpose, not per team. Separate issuing CAs for TLS server, client/mTLS, device identity, and code signing. Each can carry its own policy, validity profile, and revocation behaviour, and each becomes an independent blast-radius boundary.
- Name constraints on intermediates so a TLS-issuing CA cannot issue for namespaces it has no business in. This is one of the cheapest containment controls available and is routinely skipped.
- Deliberate naming. Subject DNs and certificate profiles that encode purpose and environment, so an operator can read a chain and know exactly what they are looking at.
+---------------------------+
| OFFLINE ROOT CA | air-gapped, in safe
| RSA-4096 / 20yr | powered on only for
| self-signed | scripted ceremonies
+-------------+-------------+
| signs intermediates only
+-------------------------+--------------------------+
| | |
+-------v--------+ +---------v---------+ +---------v---------+
| TLS Issuing CA | | mTLS / Client CA | | Device Identity CA|
| EC P-384 / 6yr | | EC P-384 / 6yr | | EC P-384 / 6yr |
| nameConstraint:| | nameConstraint: | | nameConstraint: |
| *.corp.example| | spiffe://corp/* | | dn: OU=devices |
| online, HSM | | online, HSM | | online, HSM |
+-------+--------+ +---------+---------+ +---------+---------+
| issues | issues | issues
leaf certs workload identities device certs
(ACME, 90d) (short-lived, mins-hrs) (per-device)Each intermediate failing independently is the whole point. If the device CA is compromised, you revoke and re-issue one intermediate; you do not touch TLS, and you never touch the root.
Key ceremonies and HSM operations
Root key material is the most valuable secret your organisation holds, and it should be treated with more ceremony than any production deployment. A root key generation or signing event is a scripted, witnessed, recorded ceremony. Nobody should be SSHing into a box on a Tuesday afternoon to do it.
The non-negotiables: keys are generated inside FIPS 140-2 (or 140-3) validated HSMs and never exist in plaintext outside them. Control of the key is split under m-of-n quorum, so no single individual can exercise the root. Every step is read from an approved script, executed, and attested by independent witnesses, and the HSM's own audit log of key operations is exported and retained.
A root key that one person can use alone is not a root of trust; it is a single point of betrayal wearing a certificate.
ROOT CA KEY CEREMONY — abbreviated runbook
==========================================
Quorum: m-of-n = 3-of-5 key custodians (smartcards)
Roles: Ceremony Master, Witness x2, Security Officer, Scribe
Location: access-controlled room, no network, video recorded
1. Verify tamper-evident seals on HSM and safe; log serial numbers.
2. Power on HSM; confirm FIPS mode active and firmware hash matches.
3. Assemble quorum: 3 of 5 custodians authenticate smartcards.
4. Generate root keypair IN HSM (EC P-384 or RSA-4096).
- private key marked non-exportable, sensitive.
5. Self-sign root certificate from approved profile.
6. Export public root cert + fingerprint; read SHA-256 aloud, Scribe records.
7. Sign each subordinate CSR against its named profile.
8. Export HSM audit log to signed, write-once media.
9. Re-seal HSM and custodian cards into separate safes (split custody).
10. All participants sign the ceremony record; archive with video.The theatre has a practical purpose. Two years later, when an auditor or an incident asks "who could have used this key, and can you prove what it signed?", the answer is written down, signed, and tamper-evident.
Compliance and audit: "prove it" is a first-class requirement
If your CA is publicly trusted, you do not get to mark your own homework. You operate to the CA/Browser Forum Baseline Requirements, you submit to annual WebTrust (or equivalent ETSI) external audit, and every certificate you issue is logged to Certificate Transparency where the entire internet can inspect it. Misissuance becomes a public, timestamped record, and the consequence of serious or repeated failure is removal from root programmes, which is commercial death for a public CA.
Even for a purely internal PKI, treat "prove it" as a design requirement rather than an afterthought. Internal audit will ask the same hard questions: who can issue, under what controls, with what key custody, and where is the evidence. Build the attestation in from the start:
- Attestable key custody: HSM-backed, with exported and retained logs of every key operation.
- Immutable issuance logs: an internal transparency log of every certificate issued, queryable after the fact.
- Profiles as code: certificate profiles, validity caps and EKUs version-controlled and reviewed, so policy is enforced mechanically rather than by good intentions.
The cheapest time to make a system auditable is before it issues its first certificate. The most expensive is the week after an incident.
Observability and trust SLOs
You cannot operate what you cannot see, and one of the most common causes of self-inflicted PKI outages remains the certificate nobody knew about expiring at 02:00. The foundation of operating at scale is a certificate inventory that is the single source of truth. It should be discovered rather than self-reported, and cover what exists, where it lives, who owns it, when it expires, and whether it is renewing automatically.
Alert on renewal failure, not on expiry
If your monitoring fires when a certificate is seven days from expiry, you have built a system that alerts you to your own failure after it is too late to fix calmly. The signal that matters is the renewal attempt failing, which happens days or weeks before expiry while there is still slack. Expiry alerting is a backstop, not a strategy.
From the inventory you can define and measure real trust SLOs:
- Renewal health: percentage of certificates renewing automatically and on schedule (target high, e.g. 99.9% of eligible certs renew before the 2/3 lifetime mark).
- Coverage: percentage of issued certificates under automated lifecycle vs manual. Manual is technical debt; measure it so it shrinks.
- Revocation responsiveness: time from "revoke decision" to revocation status being globally visible (CRL/OCSP propagation).
- Issuance availability: the issuing CA / ACME endpoint as a tier-one service, because if issuance is down, everything that needs a fresh certificate eventually breaks.
Incident response: the scenarios that keep you up at night
Three scenarios worry architects most, in ascending order of severity: a mass revocation event, an issuing-CA compromise, and a root compromise. The time to design the response is now, on a calm afternoon, not at the moment it happens.
Mass revocation
The Baseline Requirements give you tight windows (generally 24 hours or 5 days, depending on the reason) to revoke misissued or key-compromised certificates. If you cannot re-issue and re-deploy thousands of certificates inside that window, revocation itself becomes an outage. This is why I treat the automation from Part 3 as a safety control rather than a convenience. A fleet on ACME with short lifetimes can absorb a mass revocation; a fleet of hand-installed five-year certs cannot.
Issuing-CA and root compromise
RUNBOOK: SUSPECTED ISSUING-CA KEY COMPROMISE
============================================
SEV-1. Page PKI on-call + Security IR + comms.
CONTAIN
[ ] Disable issuance on the affected CA (revoke API creds, stop ACME).
[ ] Snapshot HSM audit logs + issuance log; preserve evidence.
[ ] Scope blast radius: query inventory for ALL certs under this CA.
DECIDE
[ ] Confirm vs suspect compromise (HSM logs, anomalous issuance?).
[ ] If confirmed: ROOT revokes the affected intermediate.
--> every leaf under it is now untrusted. This is intended.
RECOVER
[ ] Stand up replacement intermediate from offline root (ceremony).
[ ] Re-issue leaves under new intermediate via automation (ACME).
[ ] Publish updated CRL/OCSP; verify global propagation.
VERIFY & LEARN
[ ] Inventory shows 0 active certs under revoked CA.
[ ] Post-incident review; feed gaps back into CP/CPS + automation.Notice what makes that survivable. Because the hierarchy was segmented per purpose (Part 1), revoking one intermediate does not take down the others. Because issuance is automated (Part 3), re-issuing thousands of leaves is a scheduled job rather than an all-hands effort. The genuine nightmare, root compromise, is survivable only if you planned for it: a backup or successor root cross-signed or pre-distributed, the ability to push new trust anchors to your estate, and an estate built for crypto-agility (Part 4) so swapping the anchor is a managed rollout rather than a rebuild. If your only plan for root rotation is "manually touch every device," what you actually have is a future outage with a long tail.
Organisational reality
At enterprise scale the hardest problems tend to be about ownership rather than cryptography. A common failure mode is the orphaned PKI: built by a project team, inherited by nobody, quietly accruing risk until an expiry takes down production.
PKI needs a named owning team with a clear remit: custody of roots, the CP/CPS, the issuing infrastructure, and the SLOs. But that team must not become a ticket queue that every developer hates. The way to reconcile central control with developer autonomy is the golden path: self-service, automated issuance through approved profiles, where the right thing is also the easy thing.
- A developer requests identity for a workload and gets a short-lived certificate via ACME or a workload-identity API in seconds, without raising a ticket.
- The profile, validity cap, key type and naming are enforced centrally and invisibly. The platform team owns the rails; teams own their trains.
- Manual issuance still exists for edge cases but is the exception, is logged, and has friction by design, because friction is what drives people onto the golden path.
And be honest about cost and complexity. Every additional intermediate, profile and policy is something to operate, monitor and audit. Segment where there is a real blast-radius boundary, and resist doing it for tidiness. Aim for the minimum hierarchy that gives you the containment and policy separation you actually need, and not one CA more.
Closing the series: trust as living infrastructure
If there is one idea that ties these six parts together, it is this: digital trust is not a thing you build and walk away from. It is living infrastructure. A PKI is a set of promises, encoded in policy, enforced by cryptography, and kept or broken by how you operate it day after day. The certificate is only the visible artefact of that discipline.
We started by anatomising a PKI and we end by running one, and the through-line is that good architecture is the thing that makes good operations possible. The hierarchy you designed deliberately is what contains your worst day. The automation you built is what lets you survive a mass revocation inside the window. The crypto-agility you engineered is what turns a root rotation from an extinction event into a rollout. None of that is luck. It comes from decisions made early and on purpose. Build trust to be operated, instrument it so you can see it and rehearse the day it fails, and it will quietly hold up everything else you are building on top of it for years to come.
NK