The CyberArk + HSM failure drill: what the vault does when the HSM is unreachable

Once the CyberArk server key lives in a Luna HSM, you have deliberately made your privileged access system depend on a second system to function. That is a good trade in a regulated environment — but only if you know, precisely, what the vault does when that second system disappears. Most teams don’t. They know the happy path, they’ve read that “the vault needs the HSM”, and the first time they see the real behaviour is during an incident, when the room is already loud. This chapter is the failure drill: the taxonomy, the observed behaviour, the signals that separate “HSM down” from “vault down”, and a runbook structure your NOC can execute at 2am without waking the crypto custodians for the wrong reasons.

It builds directly on the previous chapter, designing Luna HSM HA for a primary + DR vault pair — the topology you chose there decides which of these failures you can actually experience.

The failure taxonomy: startup versus mid-flight

The vault has two fundamentally different relationships with the HSM, and mixing them up leads to wrong triage decisions:

• At startup, the vault must open a PKCS#11 session and access the server key (ServerKey=HSM#N in DBParm.ini) before it can do anything at all. No HSM, no start — the PrivateArk Server service fails to initialise, and the PKCS#11 provider logs show variations of “device cannot be found”. This is by design, not a bug: a vault that started without its key-encryption-key would be a vault that had the key cached somewhere it shouldn’t be.
• Mid-flight, the picture is more nuanced. A vault that is already running does not fall over the moment the HSM drops. Operations that don’t need a fresh safe-key decryption — user authentication, some reporting and administrative functions — can continue. What breaks is the crown-jewel operation: retrieving or storing credentials that require decrypting a safe key via the server key. Those fail with safe-key decryption errors while everything around them looks deceptively healthy.

This asymmetry is why “the vault is up” is a dangerous health check. A monitoring probe that logs in and calls it green will happily report a vault that can no longer hand out a single password.

What recovery looks like when the HSM comes back

In the mid-flight case, restoration is normally automatic: once HSM connectivity returns, the vault re-establishes its session and credential retrieval resumes — with the first retrieval noticeably slow while the session and key handles are rebuilt, then normal performance after that. Where automatic recovery doesn’t happen, CyberArk provides a ReconnectHSMOnErrorCodes parameter in DBParm.ini: you identify the PKCS#11 error codes appearing in vault trace logs (with debug levels CRYPT(1,2) enabled) and list them so the vault knows those errors warrant a reconnect attempt rather than a hard failure. The specific codes are environment- and version-dependent — read them out of your own trace logs rather than copying someone else’s list.

On the Luna side, remember the client has its own recovery machinery layered underneath the vault’s: in an HA group, member failover triggers after roughly a 20-second timeout, health checks run on ~10-second heartbeats, and auto-recovery (if you enabled it — it is disabled by default) re-admits returning members on a configured interval and retry count. A single HSM reboot inside a healthy HA group should therefore be invisible to CyberArk. If a one-member event was visible to users, that is a finding about your HA configuration, not about Luna.

What users and PSM sessions actually experience

Translating the taxonomy into user-visible symptoms, because that’s what the tickets will say:

• PVWA users can typically still log in and browse, then hit errors on “Show/Copy password” or “Connect” — the operations that need a safe-key decryption. Expect tickets that say “CyberArk is being weird”, not “CyberArk is down”.
• Established PSM sessions that already hold their target credentials generally keep running — the RDP/SSH stream doesn’t need the HSM. New PSM connections fail, because the connection component can’t retrieve the target account’s credential.
• CPM starts logging failed verify/change/reconcile operations. This one has teeth: a change that got as far as setting a new password on the target but couldn’t finalise vault-side state can leave accounts needing reconciliation once service returns. Check CPM logs for in-flight changes as part of recovery, not as an afterthought.
• After a full vault restart during the outage window, the failure converts from partial to total: the vault won’t start at all until the HSM path is restored. Resist the instinctive “have you tried restarting it” — a restart makes this incident strictly worse.

Monitoring signals: “HSM down” versus “vault down”

The NOC’s first decision is which system is actually failing, and the two produce distinguishable signatures if you monitor both layers:

Signal	HSM unreachable	Vault down
PrivateArk Server service	Running (until restarted)	Stopped/failed
PVWA login	Usually works	Fails
Credential retrieval	Fails with decryption/HSM errors	Fails (no vault at all)
Vault logs (italog / trace)	PKCS#11 / safe-key decryption errors	Service-level failure
NTLS reachability from vault (TCP 1792)	Down or degraded	Normal
Luna appliance (lunash ntls/syslog, SNMP)	Errors, dropped clients	Clean
Existing PSM sessions	Continue	Depends on component; new and existing activity degrades

Two cheap probes make this table operational: a synthetic transaction that actually retrieves a known test credential on a schedule (proves the whole chain, not just the login page), and independent monitoring of the Luna appliances themselves — NTLS service state, client connections, and syslog forwarding to your SIEM, which the ISM’s logging controls want anyway. On the appliance, the checks the runbook should name explicitly:

lunash:> service status ntls
lunash:> ntls information
lunash:> client list

One more signal worth naming: during an HSM outage the vault’s PADR replication to the DR site generally continues — replication is a data operation, not a fresh safe-key decryption. So a green PADR log during an HSM incident tells you your DR copy is current; it tells you nothing about whether the DR vault could start. Don’t let it be read as “DR is fine”.

The NOC runbook: detection to escalation

Structure the runbook so the night shift can execute the first three stages without any crypto knowledge:

1. Detection

• Synthetic credential retrieval fails, or HSM reachability alarm fires, or a cluster of “can’t copy password / can’t connect” tickets lands.
• Record the time. The gap between HSM loss and first user impact is evidence you’ll want later.

2. Triage — classify using the table

• Is PrivateArk Server running? Can users log in to PVWA? Does retrieval of the test credential fail with decryption errors?
• From the vault server, test the NTLS path to each HSM (TCP 1792). Check the Luna client’s view (vtl verify / lunacm slot visibility) if trained to.
• One HSM or all of them? One member of an HA group down is a hardware ticket, not an incident. All members unreachable is a network story until proven otherwise.

3. Decision points — the ones that go wrong under pressure

• Do not restart the vault service. A running vault is partially alive; a restarted vault with no HSM is fully dead. Make this a written, named rule.
• Do not fail over to DR reflexively. If the cause is the network between vault and HSMs, the DR vault may inherit the same problem — and if your topology is per-site HSMs, first confirm the DR partitions hold the current server key.
• Decide the break-glass threshold in advance: at what duration of credential-retrieval outage do you invoke the offline break-glass process for critical target systems? That number belongs in the runbook, not in the incident bridge.

4. Escalation

• Network path issues → network on-call. Appliance issues (NTLS down, appliance unresponsive) → HSM custodians/platform team, then Thales support. Vault-side errors persisting after HSM restoration → CyberArk admin, checking ReconnectHSMOnErrorCodes and trace logs before any restart.
• After recovery: run CPM reconciliation checks, review which accounts had in-flight changes, and write the timeline down while it’s fresh.

A tabletop-exercise template accompanies this chapter — a scripted version of this scenario you can run with the NOC and the CyberArk team in under an hour. The series hub will link it when it’s published.

How this plays out: a composite scenario

A pattern we see repeatedly in regulated environments, assembled here as one composite. A firewall change window tightens rules between the server zone and the “security appliances” zone; TCP 1792 from the vault to the HSMs is not on the approved list because nobody knew the vault was an HSM client. Nothing happens — for three weeks, because the vault’s NTLS sessions were already established and nothing forced a reconnect. Then a Sunday patch window reboots the vault server. The vault doesn’t come back. The on-call engineer sees a service that won’t start, assumes vault corruption, and burns two hours in CyberArk logs before anyone reads the PKCS#11 provider log with its “device cannot be found” line. Total outage: most of a business day, root cause: a firewall rule, detection gap: the monitoring only ever watched the vault. Every element of that story is boring; the combination is a sev-1. The runbook above exists to make the combination boring too.

---

Where this leaves you

You now have a system whose failure modes are drilled and documented — and a harder problem waiting at audit: none of it stands up if you can’t show who holds the partition secrets, how the server key was generated, and that no single person could do both. Key ceremony and separation of duties are the next chapter. For the CyberArk-side architecture underneath all of this, the CorePAS implementation guide is the companion read, and the full series lives at The Luna Field Guide.