An issuing CA on a Luna HSM HA group: what happens when a member drops mid-request

The root CA gets all the ceremony — the witnessed key generation, the safe, the offline appliance. But the root signs a handful of certificates a decade. The issuing CA signs thousands a week, and every one of those signatures is a call to a Luna partition. When teams in federal and regulated-PII environments put ADCS on Luna hardware, the issuing CA is where availability engineering actually matters — and it is also where the single most damaging misconfiguration in the whole ADCS-on-Luna stack lives: a CA key that was generated against one physical HSM instead of the HA group’s virtual slot, and therefore exists on exactly one appliance. This chapter covers why the issuing tier carries the availability burden, how that trap happens, what a member failure looks like from the CA’s point of view mid-request, and how to prove — not assume — that your CA key is on every member.

Why the issuing CA is the availability-critical tier

An offline root can be down for months and nobody notices, because it is supposed to be down — its availability problem is really a recoverability problem, which is why we framed the root chapter around proving the restore rather than uptime. The issuing CA inverts that. It sits in the hot path of the estate: autoenrolment for machine and user certificates, 802.1X and Wi-Fi authentication, smartcard logon, web server TLS, NDES for devices. In an Australian government context, the certificate services underpinning ISM-aligned authentication controls are exactly the kind of dependency an IRAP assessment expects to see availability evidence for.

Two properties make the issuing tier unforgiving:

• Every issuance is an HSM operation. The CA’s signing key never leaves the partition, so every certificate signature, every CRL signature, and every OCSP response (if the responder’s key is also HSM-held) is a network call to a Luna appliance. HSM availability is CA availability.
• Failure is invisible until demand arrives. A CA that cannot sign does not page anyone at 2 am. It fails when the next enrolment arrives — or worse, when the CRL expires and every relying party in the estate starts failing revocation checks at once.

This is precisely the workload Luna HA groups exist for: two or more appliances behind one virtual slot, load-balanced signing, automatic failover. When it is built correctly, an issuing CA on an HA group rides through a member failure with a pause measured in seconds. The rest of this chapter is about the ways “built correctly” quietly fails to be true.

The trap: a CA key that exists on one HSM

Recall from the HA groups chapter that Luna HA is a client-side construct: the client presents a virtual slot backed by real partitions, and a key generated in the virtual slot is created on the primary member and cloned to the others. The critical corollary: a key generated in a physical member slot is just a key on one HSM. Nothing replicates it. The group does not know it exists.

Here is how that happens to an issuing CA. When you install ADCS with the SafeNet Key Storage Provider, the CA key is generated in whichever Luna slot the KSP was registered against. The KSP registration (via KspConfig.exe, for both the service account context and NT AUTHORITY\SYSTEM — the subject of the KSP gotcha chapter) happens on a client that may be able to see three slots: member A’s partition, member B’s partition, and the HA virtual slot. If HA-only mode has not been enabled on the CA server’s Luna client, the physical slots are sitting right there, usually with lower slot numbers than the virtual slot — and a slot registration made against a physical partition looks identical, works identically, and generates a CA key that will never exist anywhere but that one appliance.

Thales’s own ADCS integration guidance is blunt about this: HA-only must be enabled for failover to work, so that all calls route through the virtual slot and fail over automatically when a member drops. Without it, three failure modes are on the table:

• The key never replicates. The CA runs perfectly — issuing, signing CRLs, passing health checks — for years, against one physical HSM. The second appliance is a very expensive space heater holding an empty or stale partition.
• Failover never triggers. Even if the same key somehow exists on both members, a KSP bound to a physical slot has no failover logic. The Luna HA machinery only engages for calls made against the virtual slot.
• Synchronisation can destroy it. Objects created directly in a physical member slot sit outside the group’s object table, and Thales documents that a subsequent hagroup synchronize can overturn direct physical-slot changes. An operator “tidying up” the group can delete the CA key.

The dangerous property, as ever with Luna HA, is that nothing is visibly wrong. The virtual slot and the physical slot both answer. The CA issues either way. The difference only becomes observable when a member fails — which is the one moment you cannot afford to be finding out.

What a member failure looks like mid-request

Assume the healthy case: CA key generated in the virtual slot, present on both members, HA-only enabled. A member drops — hardware fault, switch failure, someone’s change window. What does the CA experience?

The Luna client detects the failure on its connectivity and command timeouts — roughly 20 seconds by default, tunable via ProbeTimeout and related settings in the client configuration file, as covered in the HA chapter. During that window, signing calls dispatched to the failed member stall. Once the client declares the member dead, pending operations are rescheduled onto the surviving member and new operations route there directly. From the CA’s perspective:

• Most requests just go slow. Requests that arrive during the detection window queue behind the stalled calls and complete once failover finishes. Enrolment clients see a pause of tens of seconds, not an error.
• Operations mid-flight on the failed member can surface an error. A signing call that was physically executing on the dead appliance when it died may return a failure to the KSP rather than being silently replayed. For that individual certificate request, the CA reports an error and the request fails.
• The retry burden lands on the enrolment layer — and mostly it copes. Windows autoenrolment treats a failed enrolment as retryable: the client tries again on the next autoenrolment pulse (triggered at boot, logon, policy refresh, and on the periodic timer — roughly every eight hours by default). A human driving certreq or the MMC sees the failure and resubmits. The population that genuinely suffers is bespoke integrations that call the CA once, treat an error as fatal, and never retry — find those before the drill, not during it.

Contrast the unhealthy case. If the CA key lives only on the failed member, the surviving member cannot help — the client fails over to a partition that does not hold the key. Every signature attempt now fails hard. The CA service may keep running while every request errors, or hang and stop, depending on how the KSP surfaces the failure. And no amount of restarting certsvc fixes it, because the problem is not the service — it is that the key material is inside a dead appliance. Recovery is now a backup-restore or key-clone exercise under incident pressure, assuming a backup exists.

The autoenrolment surge after an outage

A subtlety teams miss when sizing the issuing tier: certificate demand is not smooth, and outages make it spiky. If the CA is down for a period — an extended HSM incident, or the single-member scenario above — enrolments do not disappear; they defer. Every machine whose renewal window opened during the outage retries on its next autoenrolment pulse. When the CA comes back, it faces the accumulated backlog roughly simultaneously: hundreds or thousands of requests, each one an HSM signing operation, arriving on top of the CRL signing and normal traffic that also resumed.

An HA group absorbs this well — load-balanced signing across members is exactly what the architecture is for. But it is another reason the surviving-member case matters: after a member failure, your surge capacity is halved at precisely the moment a backlog may be forming. If your capacity planning assumed both members for peak issuance, a single member limping through a post-outage surge is the day you find out. Plan the tier so that one member can carry peak-plus-backlog, and treat the second as the redundancy it is.

Verifying the CA key exists on every member

“The HA group is synchronised” and “my CA key is on both members” are different claims, and only the second one saves you during an incident. Verification is direct: list the key objects in each physical member partition and confirm the CA key appears in all of them.

The tool is cmu, the certificate management utility shipped with the Luna client. Because HA-only mode (correctly) hides physical slots on the CA server, run the check from an admin workstation registered to both partitions without HA-only, or temporarily re-expose slots in a maintenance window. Against each member’s slot:

cmu list -slot=0 -display=class,keyType,label -class=private

class=privateKey     keyType=RSA     label=ISSUING-CA-01

cmu list -slot=1 -display=class,keyType,label -class=private

class=privateKey     keyType=RSA     label=ISSUING-CA-01

The CA’s key label should appear in every member partition, with matching counts of private key objects across members. A label present in slot 0 and absent in slot 1 is the single-member trap, caught on your schedule instead of the HSM’s. Pair the per-member check with the group-level and CA-level views: hagroup listgroups in LunaCM should show all members alive and Needs sync: no, and on the CA itself, certutil -verifykeys confirms the CA can actually open and use its key through the KSP. Run all three on a schedule and after every change to the group — the how-to companion wires this into a monitoring checklist.

How this plays out: a composite field scenario

The following is an anonymised composite of a pattern that recurs in community reports and regulated environments — no single organisation, but nothing in it is invented.

An agency builds a two-member HA group for its enterprise issuing CA. The HA group is real: correctly domained, synchronised, sign-off in the design doc. But the ADCS build was done by a different team, months later, from a runbook written against a single-HSM lab. The engineer registered the first slot KspConfig offered — member A’s physical partition — and installed the CA. Everything worked. certutil -verifykeys passed. The project closed.

Three years on, member A is powered down in a rack consolidation, on the strength of “we have HA”. The CA stops issuing instantly. Autoenrolment failures accumulate quietly for two days — nobody watches pending-request queues — until the CRL hits its expiry window, revocation checking fails estate-wide, and 802.1X starts refusing devices. The incident call establishes, in this order: the HA group is healthy; the surviving member is fine; the CA key is not on it and never was. Recovery meant powering the decommissioned appliance back up to clone the key properly into the group — possible only because the hardware still existed in a loading dock. The post-incident review found the whole event had been detectable for three years with one cmu list per member.

What good looks like for the issuing tier

Condensing the above into the standard we hold issuing CAs to in ISM-aligned environments:

• CA key generated into the HA virtual slot, never a physical member slot — and HA-only enabled on the CA server before ADCS is installed, so the physical slots were never available to get it wrong.
• Per-member key presence verified with cmu list at build time, after every member replacement, and on a recurring schedule — as evidence, not assumption.
• A member-failure drill run against the CA specifically: issue certificates while a member is pulled, observe the pause, confirm autoenrolment retries cover the mid-flight failures.
• Capacity planned so one member carries peak issuance plus a post-outage backlog.
• CRL publication monitored with alerting well inside the expiry window — CRL expiry is how an HSM problem becomes an estate-wide authentication outage.

The how-to companion walks the full build end to end, including the drill. For the broader platform context — where the client, partition, and appliance boundaries sit — the foundation chapter remains our Luna HSM design and operational guide.

---

Where this leaves you

Your PKI’s availability tier is now defensible: key on every member, failover proven, verification scheduled. The next keys demanding this treatment are not CA keys at all — since the CA/Browser Forum mandated hardware protection for publicly-trusted code signing keys, every team that signs software has been pushed onto HSMs, usually with none of the operational discipline above. Code signing on Luna hardware, and what the mandate actually requires, is the next chapter. The full series lives at The Luna Field Guide.