HSM key ceremonies and separation of duties: who holds which key

A key ceremony has a reputation problem. Say the words and people picture robes and candles: a room full of witnesses watching someone press buttons on a PED, everyone signing a form nobody will read again. In an environment holding identity-verification data or police-check-class PII, that picture is exactly wrong. The ceremony is not theatre performed for auditors — it is the one moment where your organisation decides, physically and irreversibly, who can administer the HSM, who can use its keys, who can clone them, and who can watch. Every one of those decisions is embodied in a small plastic key handed to a specific person, and most of them cannot be quietly revised later.

This chapter is about getting those decisions right: the Luna PED key colour taxonomy and what each colour actually unlocks, how to design M-of-N splits that survive staff churn, how HSM key custody must be mapped against CyberArk vault recovery-key custody so no individual ends up holding both sides of the trust boundary, and what a ceremony record has to contain to work as assessor evidence under the ISM and IRAP. It assumes the platform grounding from our Luna HSM design and operational guide.

Why the ceremony is a control, not theatre

On a PED-authenticated (multifactor quorum) Luna HSM, administrative and cryptographic roles are not passwords in a spreadsheet — they are secrets imprinted onto physical PED keys at initialisation. Whoever walks out of the room with a given key holds that role, full stop. There is no help-desk reset, no break-glass account, no “IT can just fix it”. That physicality is the entire point: it converts separation of duties from a policy statement into a property of matter. Two roles are separated if, and only if, their keys sit in different hands and different safes.

It also means the ceremony is where your future audit posture is fixed. The Australian Government ISM requires that cryptographic key management processes, and supporting procedures, are developed, implemented and maintained (ISM-0507) — covering generation, distribution, storage, access, recovery and destruction. An IRAP assessor testing that control does not want your policy document; they want evidence that the day the keys came into existence, the right people held them, the wrong people did not, and someone wrote it down. The ceremony record is that evidence. Run the ceremony without the record and you have done the work but cannot prove it — which, at assessment time, is indistinguishable from not having done it.

The PED key colour taxonomy: what each key unlocks

Thales colour-codes the PED keys by role. The colours are consistent across the Luna 7 line, though role naming has shifted between releases — current documentation calls PED authentication “multifactor quorum” — so verify terminology against your appliance’s firmware documentation. The taxonomy:

• Blue — Security Officer (SO). The administrator of the HSM (and, at partition level, the Partition SO). The blue key initialises, sets policies, creates and deletes partitions, and applies firmware. Critically, the SO administers the container but cannot use the keys inside it.
• Red — cloning domain. Not a login at all, but a shared secret defining which HSMs and partitions may clone key material to each other — the boundary for backup, restore and HA. It is set at initialisation and can never be changed afterwards; we covered the consequences in the cloning-domain chapter. The red key is the most commonly under-guarded key in the set, and it is the one that decides where your keys can travel.
• Black — Crypto Officer (CO). The working role: creates and manages keys within a partition and performs cryptographic operations. This is the key your applications’ partition ultimately answers to, and the one activation caches for unattended services.
• Gray — Crypto User (CU). A restricted-use role: can use existing keys for cryptographic operations but not create, delete or manage them. The CO/CU split lets an application run with use-only rights while key management stays with a human quorum.
• Orange — Remote PED vector (RPV). Authenticates a Remote PED connection, letting a PED in one secure room serve an HSM in another. Whoever holds orange keys can bring PED authentication to the appliance across the network — which makes it a custody decision, not a convenience accessory.
• White — Auditor. A deliberately independent role that manages the HSM’s audit logging. The Auditor cannot touch keys, and the SO cannot manage the logs — so the person who administers the HSM cannot also erase the record of having done so.

The minimum viable custody model falls straight out of the taxonomy: blue, black and white in three different hands, red with a fourth custodian who is not part of day-to-day operations, orange treated with the same seriousness as blue. An organisation that hands blue and black to the same engineer has collapsed the distinction between “administers the vault” and “uses what is in it” — the exact distinction the hardware exists to enforce.

Mapping HSM custody against CyberArk recovery custody

If your Luna estate protects a CyberArk Digital Vault — the integration we examined in the server-key dependency-inversion chapter — then the HSM keyholders are only half of the custody picture. The vault has its own crown-jewel material: the recovery key pair, with the recovery private key (recprv.key, distributed on the Master media) enabling the Master user to log on and open every safe in a recovery scenario. CyberArk’s guidance is that this material lives offline, on removable media, in a physical safe, never on the vault’s own filesystem.

Now put the two custody lists side by side and check one property: no individual appears on both. The person (or quorum) who can authenticate the Luna partition serving the vault’s server key controls the root of the vault’s encryption hierarchy. The person who holds the vault recovery material can become the Master user. Either half alone is a strong position hedged by the other; both halves in one pair of hands is a single insider who can operate the key-protection layer and the data-recovery layer of your PAM estate. In smaller security teams this failure happens by default, not by malice — the same senior engineer who ran the HSM ceremony also built the vault, so both sets of material end up in the one safe that engineer controls.

Material	What it unlocks	Must not be co-held with
Blue PED key (SO)	HSM/partition administration, policy change	Black CO key; vault recovery media
Black PED key (CO)	Key management and use in the vault’s partition	Vault recovery media; White Auditor key
Red domain key	Cloning, backup and restore boundary	Sole custody by any operational keyholder
White Auditor key	HSM audit log control	Blue or black keys
CyberArk recovery private key + Master credentials	Master logon, full safe recovery	Any Luna keyholder for the vault’s partition

M-of-N splits that survive staff churn

Luna lets you split any role’s secret across up to sixteen PED keys and require a quorum — M of N — to authenticate. Used well, M of N means no single person can exercise a sensitive role alone. Used badly, it is how organisations lock themselves out of their own HSM three years later. Thales’s own guidance flags both failure modes: M equal to N means one lost key, one resignation or one flooded safe blocks the role forever; M equal to one is a split in name only.

The splits that survive contact with HR reality share a few properties:

• Headroom over the quorum. Three-of-five tolerates two simultaneous absences — a resignation and a holiday — without an emergency. Two-of-three is the sensible floor for smaller teams; two-of-two is a resignation away from a crisis.
• Custody assigned to positions, not people. The keyholder register names roles (“Security Operations Lead, split 2 of 5”) with the incumbent recorded against the role, so a departure triggers a handover, witnessed and signed, rather than a redesign.
• Geography that matches your DR plan. If DR requires authenticating an HSM in another city, a quorum’s worth of keyholders — or a Remote PED arrangement with properly held orange keys — must exist for that site. A 3-of-5 split where all five work in one office fails with the office.
• A tested assembly drill. Once or twice a year, actually convene a quorum and authenticate against a non-production partition. The drill verifies the keys, the register, the safe access and the people — every element that quietly rots between ceremonies.

One asymmetry deserves respect: role credentials can be changed and keys re-imprinted at a later ceremony, but the red domain secret is permanent. Plan the domain key’s custody — including duplicates in separately controlled safes — as if you will still need it in fifteen years, because you will.

The ceremony record as assessor evidence

An IRAP assessor testing the ISM’s key-management and privileged-access expectations will ask three questions: who can do what to your keys, how do you know, and can you prove it has been true since day one. The ceremony record answers all three, provided it captures:

• A scripted agenda, followed and initialled step by step — the script is written before the ceremony and deviations are recorded, not improvised around.
• An attendee register with roles: who performed, who witnessed, who held which key at the end. Witnesses should include someone independent of the operating team.
• A key inventory: each PED key’s colour, label, serial, split number, custodian and safe location, plus tamper-evident bag serials for stored keys.
• The technical state produced: HSM serial and firmware, partition names, and the policy set as configured — export the policy listing from the appliance and attach it, so the record shows the configuration as built, not as remembered.
• Signatures, dates, and a filing location the assessor can be walked to. A record nobody can find fails the same way a record nobody wrote does.

Keep the register living: every custody transfer, safe audit, duplicate creation and assembly drill appends to it. The initial ceremony proves the control existed once; the register proves it still exists.

How separation fails in the field

A composite pattern, assembled from community war stories and support-forum threads rather than any single engagement — but it will be familiar to anyone who has inherited an HSM estate in a regulated environment.

The HSM went in four years ago, delivered by a contractor under deadline. The ceremony happened, in the sense that keys were imprinted: all of them by the one engineer, all labelled in that engineer’s handwriting, all placed in the one comms-room safe — blue, black, red, orange together, because separate safes felt like a problem for later. No M of N; one key per role, because the PED prompts default that way. The record is a photo of the keys on a desk. The engineer’s contract ended in year two. In year four, an IRAP assessment asks who holds the SO role, and the honest answer is “whoever opens that safe” — currently eleven people with the door code. The findings write themselves: no demonstrable separation between administration and use, no custody register, no evidence the domain key even matches the production HSMs — and confirming it does would mean a restore test nobody has ever run. None of this is a hardware weakness. Every control the HSM offered was present and unused, and the remediation — a full re-ceremony, custody redesign and register rebuild, done live under an assessor’s gaze — costs ten times what an afternoon of custody planning would have cost on day one.

Where this leaves you

With the colour taxonomy understood, custody mapped across both the HSM and vault-recovery sides, and a register an assessor can actually test, you have separation of duties as a working control rather than a diagram. The next place all of this gets tested for real is PKI: an offline root CA’s key ceremony compresses every custody, quorum and evidence decision in this chapter into one high-stakes day — and adds the question of whether you can ever prove the root restores. That is where the series goes next. The full series lives at The Luna Field Guide.