An offline root CA on a Luna HSM: the ceremony matters less than the restore

Every offline root CA build ends the same way: a conference room, a printed script, two or three witnesses, a laptop that has never touched a network, and a signing ceremony that produces a root certificate, a stack of signed records, and a feeling of having done things properly. Almost none of them end with the one test that determines whether the root survives the decade: restoring the root’s private key onto a different HSM and proving it still signs. This chapter argues that for a root CA on a Thales Luna HSM, the ceremony is the easy 90 per cent — and the restore proof is the 10 per cent that actually matters.

The two-tier recap, kept short

You know the model: an offline root CA that signs only the issuing CA certificates and its own CRL, and one or more online issuing CAs that do the daily work. The root stays powered off in a safe, comes out once or twice a year to publish a CRL or sign a new subordinate, and its private key is the single object whose compromise or loss undoes the entire hierarchy. Loss is the half people underweight. A compromised root is a catastrophe you can at least respond to — revoke, rebuild, redeploy trust. A lost root key is quieter and slower: the hierarchy keeps working right up until the root CRL expires or an issuing CA certificate needs renewal, and then nothing below it can be extended, ever. You are rebuilding the PKI from scratch, redistributing trust anchors to every device and system that pinned the old root.

That is why the root key goes in an HSM in the first place — and in Australian federal and regulated environments, why the ISM’s cryptography controls push CA keys into hardware. But putting the key in a Luna changes the failure modes, and most ceremony scripts are written as if it didn’t.

What a Luna HSM changes about the ceremony

On a software-key root CA, “backup” means copying a PKCS#12 file. Terrible for security, trivially testable — anyone can open a .p12 on a second machine and check the key is inside. On a Luna, the private key is generated inside the HSM, marked non-exportable, and never exists in plaintext outside the hardware. The only sanctioned way to get a copy is cloning: a Luna-to-Luna transfer in which objects move encrypted under secrets derived from the partition’s cloning domain, to a Luna Backup HSM or another Luna partition that shares that domain.

Three consequences follow, and each one belongs in your ceremony script:

• Your backup is not a file, it is a device. A Luna Backup HSM holding a backup partition, stored in a safe. You cannot inspect it with a text editor, mount it, or “just check” the key is there. The only way to read it is to restore it.
• Your backup is only as good as a decision made at initialisation. The cloning domain is set once, when the partition is initialised, and can never be changed — we covered why in the cloning domains chapter. If the root CA partition and the backup were domained correctly, restore works. If they weren’t, no credential, no support contract, and no amount of seniority will ever bring that key back.
• Restore requires people, not just hardware. A restore needs the backup partition’s credentials, the cloning domain secret — a recorded string on password-authenticated HSMs, a red PED key (possibly split M of N) on PED-authenticated ones — and, on the receiving side, a partition initialised into the same domain. Every one of those is a custody question. If the domain is a red key split three-of-five and two holders have left the organisation without a handover, your backup is intact and unreachable at the same time.

None of this is exotic. It is standard Luna behaviour, documented at length. What is rare is a ceremony script that treats these as things to prove rather than things to assume.

The ceremony everyone performs, and the step almost everyone skips

A typical root CA key ceremony script has the shape you’d expect: roles assigned, witnesses present, HSM initialised, partition created, ADCS installed, key generated in hardware, root certificate signed, partition backed up to the Backup HSM, credentials sealed into tamper-evident envelopes, everything signed and dated, devices into safes. Read carefully, though, and the backup step almost always says something like: “Run the partition backup. Confirm the command completes successfully. Record the backup partition name.”

That step proves exactly one thing: on that day, with those people and those credentials in the room, the backup command exited cleanly. It does not prove the backup can be restored. It does not prove the restored key can sign. It does not prove that anyone other than the people in the room — several of whom will have changed jobs before the backup is ever needed — can execute the recovery. A backup you have never restored is not a backup; it is a hope with a serial number.

The asymmetry that makes this dangerous is temporal. The ceremony’s mistakes are invisible on the day. Everything works: the HSM is healthy, the credentials are fresh, the domain secret was typed minutes ago. The backup’s job is to work in six or ten years, on different hardware, operated by different people, and by then the ceremony record is the only witness left. If the record says “backup completed” rather than “restore proven”, you find out which one you have at the worst possible moment.

What “proving the restore” actually means

Proving the restore is a concrete, scriptable procedure, and it belongs inside the ceremony itself — same room, same witnesses, same record. Three parts:

• Restore to a second HSM. Not the one that generated the key. Take the Backup HSM you just wrote, connect it to a second Luna partition — a test partition, a spare appliance, even a partition you will factory-reset afterwards — initialised into the same cloning domain, and run the restore. This exercises the entire chain: the backup media, the domain match, the credentials, and the procedure itself.
• Sign something with the restored key. Presence of a key object is not proof of a usable key. Perform a real signature with the restored private key and verify it against the root certificate’s public key — signing a test CRL is the natural choice for a root CA, because CRL signing is precisely the operation you will need in an emergency. If the restored key signs a CRL that verifies, your backup is real.
• Document it as a first-class ceremony artefact. The record should name the second HSM, the backup partition restored, who presented which credentials, the object count restored, the test signature performed and its verification result, and the disposal of the test partition afterwards (restored copies of a root key must be destroyed, witnessed, and recorded — you have deliberately created a second instance of the most sensitive key you own, and the proof is not finished until that instance is gone).

Then make it a rhythm, not a one-off. A restore proof at the ceremony validates the day-one state; a periodic re-proof — annually, or aligned to the root’s CRL publication schedule, when the hardware is already out of the safe — validates that the backup, the credentials, and critically the people can still execute recovery. Regulated environments will recognise this shape: it is the same logic the ISM applies to backup and recovery testing generally, applied to the one key where a failed recovery is unrecoverable.

Custody: the restore is a people problem wearing a hardware costume

Everything above assumes the right people can assemble the right secrets a decade after the ceremony. That is a separation-of-duties design question — who holds the backup device, who holds the domain secret, who holds the partition credentials, and how M-of-N quorums are constituted so that no individual can restore alone but no two departures can lock you out. We treat that in depth in the key ceremony and separation of duties chapter; the point here is narrower. Your custody design is only testable through a restore. A custody register can look immaculate on paper while the actual envelopes hold a mistyped domain string, and the only procedure that ever checks the paper against reality is the one most organisations never run.

How this fails in the field

A composite scenario, assembled from patterns that recur in regulated deployments. A department stands up a two-tier PKI: offline root on a Luna, ceremony professionally run, backup taken to a Luna Backup HSM, everything sealed and safed. Six years later the root CA’s HSM fails — a power event, a dead appliance, it barely matters. No immediate alarm: the issuing CAs keep issuing. The root CRL, though, expires in a few months, and now there is a deadline.

The recovery team — none of whom attended the original ceremony — retrieves the Backup HSM, initialises a partition on a new appliance, and runs the restore. It fails with a domain error. The ceremony record says a domain was set; it does not say which artefact in which envelope holds it, and the string in the envelope labelled “HSM domain” turns out to be the appliance-level domain, not the partition’s cloning domain — a distinction nobody in the room understood in year one, because the backup was never restored in year one. In the variant that stings most, the failure is even simpler: the original team, wary of writing secrets down, initialised the backup with a domain “everyone knew” — and everyone has left. The backup hardware is perfectly healthy. Every object on it is perfectly encrypted. It might as well be a brick.

This is not a hypothetical failure class. Publicly documented incidents exist — most famously, an early European national health-card PKI whose root CA HSM failed and whose backup could not be restored, forcing a rebuild of the hierarchy. And on the day it happens there is no partial credit: a root CA restore either works completely or the PKI is being rebuilt, with every trust store, every device enrolment, and every dependent system in scope. The ninety minutes a restore proof would have added to the original ceremony is the cheapest insurance in the entire PKI programme. For the broader design context around Luna estates — HA, policies, client architecture — see the comprehensive Luna HSM design and operational guide; the companion to this chapter is a full end-to-end root CA ceremony walkthrough on ADCS and Luna, with the restore proof built in where it belongs.

Where this leaves you

If your root’s ceremony record says “backup completed” and nothing more, you now have a scheduled task: retrieve the media, restore to a second HSM, sign a test CRL, and write it down — before the day the schedule is set by an expiring CRL instead of by you. And once the offline root is genuinely provable, the next trap is on the online side: ADCS talking to a Luna through the Luna KSP has a service-account registration quirk that quietly breaks CAs on reboot, and it is the subject of the next chapter. The full series lives at The Luna Field Guide.