Luna HSM HA groups: what actually replicates — and what silently doesn’t

Ask a platform team whether their Luna HSMs are highly available and you will almost always get a yes. Ask when they last pulled a member out of the group and watched what the application did, and the room goes quiet. In regulated environments — federal identity platforms, police-check-class PII stores, defence systems working to the ISM — we see the same pattern over and over: two HSMs, an HA group configured on day one, and no evidence since then that the keys on member two match the keys on member one. This chapter is about what Luna HA actually promises, what it replicates, and the specific ways it fails silently if you treat it as fit-and-forget.

HA is a client-side construct, not a cluster

The first thing to internalise: Luna Network HSMs do not cluster. There is no replication protocol running between the appliances, no quorum, no heartbeat between HSM A and HSM B. The individual partitions are unaware they are part of a group at all.

HA is implemented entirely in the Luna HSM Client. The client presents a virtual slot to your application — a synthetic PKCS#11 slot backed by two or more real partitions on separate appliances. The client maintains a virtual object table, translates virtual key handles to physical ones, and decides which member services each call. When you generate a key “in the HA group”, the client generates it on the primary member and then clones it to the others.

This architecture has three consequences that drive everything else in this chapter:

• Every client is its own HA controller. Group configuration lives in each client’s Chrystoki.conf / crystoki.ini, not on the appliances. Two application servers can have different HA settings against the same pair of HSMs — including one with auto-recovery enabled and one without.
• Replication is cloning. The same cloning mechanism used for backup is used to copy objects between members. Anything that cannot be cloned cannot be replicated.
• The appliances cannot tell you the group is healthy. There is nothing on the HSM to alert on. Health is a client-side question, answered with client-side tooling.

If you are new to the platform, the architecture chapter of our Luna HSM design and operational guide covers where the client, appliance, and partition boundaries sit.

What replicates — and the settings that decide it

In the default configuration, replication is broad: when your application creates an object in the virtual slot, the client replicates it to all active members immediately, and deletions propagate the same way. Session objects are replicated too, not just token objects. So far, so reassuring.

The exceptions are where production incidents live:

Configuration / attribute	Replication behaviour
Cloning mode (default)	Symmetric and asymmetric keys clone to all members on creation. The full-replication case.
Key Export mode partition	Only symmetric keys replicate. Asymmetric private keys do not — your application must manage them itself. A partition set up for key export quietly halves what HA protects.
Scalable Key Storage (SKS)	Only the symmetric master key (SMK) is cloned; key material lives in encrypted blobs your application stores. HA protects the SMK, not your blob store.
Non-cloneable keys (HSS and similar)	Cannot be copied between members at all — unsupported in HA groups. A key that cannot be cloned exists on exactly one HSM, whatever the slot name says.
Mixed FIPS / non-FIPS members	On Luna HSM Client 10.4.0+, synchronising a group with mixed FIPS and non-FIPS partitions fails to replicate non-FIPS-approved keys to the FIPS members. LunaCM reports an error and syncs everything else — easy to miss in a script.

The dangerous property all of these share: the virtual slot keeps working. Your application sees its keys, operations succeed, and the fact that a given key exists on only one physical member is invisible — until that member is the one that fails.

The cloning domain prerequisite you set months ago

Because HA replication is cloning, every member partition must share a cloning domain. On firmware before 7.8.0 that means all members initialised with the same domain, full stop; from firmware 7.8.0 with client 10.5.0+, Extended Domain Management allows a partition up to three domains, and a member needs at least one in common with the primary.

The catch is sequencing: the domain was fixed at partition initialisation, possibly months before anyone typed hagroup creategroup. If your DR partition was initialised by a different team with a different domain secret, you will discover it at HA-group build time — or worse, when you try to add a replacement member during an incident. We covered why this decision is effectively permanent in Luna HSM cloning domains: the decision you can’t undo; HA groups are the place that decision comes home.

Load balancing and failover are different promises

Teams often conflate the two. The HA client does both, but they behave differently and fail differently.

Load balancing applies to cryptographic operations — sign, verify, encrypt, decrypt. The client sends each request to the least-busy active member. Key management operations are deliberately not load-balanced: creates and deletes execute on one member (new keys always on the primary) and are then replicated, so state changes have a single point of origin.

Failover is the client’s reaction to a member disappearing. It watches network connectivity and command completion on roughly 20-second timeouts (tunable via ProbeTimeout and related settings in the client configuration file, and version-dependent — check your client’s documentation). When a member drops, pending operations are rescheduled onto surviving members transparently. Two caveats matter in practice: an operation that was mid-flight on the failed member can surface an error your application must retry, and the group stays up only while at least one member is reachable — a client-side network fault takes out the whole virtual slot regardless of how many HSMs you own.

The operational consequence: benchmarks that prove load balancing works prove nothing about failover. Only a drill does that.

Auto-recovery, retry counts, and the member that quietly gave up

Here is the setting that causes more silent HA decay than any other: auto-recovery is disabled by default. Out of the box, a member that fails stays failed until a human runs a manual recovery in LunaCM. The relevant knobs:

lunacm:> hagroup retry -count -1
        HA Auto Recovery Count has been set to -1

lunacm:> hagroup interval -interval 120
        HA Auto Recovery Interval has been set to 120 seconds.

A retry count of 0 (the default) disables auto-recovery; -1 retries without limit; a positive number (up to 500) sets a fixed budget of attempts, spaced by the interval (60–1200 seconds, default 60). Setting the retry count is what enables auto-recovery. The failure mode with a finite count: a member drops during a long network change, the retry budget is exhausted before connectivity returns, and the member is now permanently out of the group until someone runs hagroup recover — which nobody does, because nobody is watching.

Recovery mode is a separate setting: activeBasic runs a background recovery thread that detects returning members, recovers them, and resynchronises content; activeEnhanced does the same and additionally restores sessions and login state after a total-group outage. Remember that these are per-client settings — every application server that talks to the group needs them, not just the box where the HSM team happened to run LunaCM.

Standby members: synchronised but idle

A standby member (set with hagroup addstandby) receives every object replication but performs no cryptographic operations unless all active members go offline. It is the right shape for a DR-site HSM: always current, never adding cross-site latency to the production traffic path, activated automatically in a genuine blackout.

Two things standby is not. It is not a backup with history — an HA group holds exactly one generation of material, and a corrupt or wrongly deleted key replicates to the standby as faithfully as a good one. And it is not free capacity — if your throughput planning counted the standby, your planning is wrong on the day it matters least.

How “we have HA” fails silently: a composite field scenario

The following is an anonymised composite of a pattern that appears repeatedly in community reports and in regulated environments we work with — no single organisation, but every element is real.

An agency runs an identity-verification platform against a two-member HA group. Built correctly two years ago: same cloning domain, group synchronised, sign-off obtained. Since then: an operational script was written that talks to a physical member slot rather than the virtual slot, because HA-only mode was never enabled and slot 0 “just worked”. Keys created that way exist on one member only — and worse, the next hagroup synchronize can overturn direct physical-slot changes entirely. Separately, member two dropped off the network for an afternoon during a switch migration; auto-recovery was never configured, so it never rejoined, and the client happily served everything from member one. The HA log that would have recorded the failover was never enabled — it is off by default.

Eighteen months later, member one takes a hardware fault. Failover “works” exactly as designed: the client fails over to member two, which is missing a quarter of the key estate and is stale on several more. The outage isn’t the HSM failure — it’s the discovery, under incident conditions, of every replication gap accumulated since the group was built. Nothing malfunctioned. Every one of those gaps was detectable for eighteen months by anyone who looked.

Monitoring HA health properly

Everything in that scenario is visible to a client that asks. The minimum standard we recommend for ISM-aligned environments:

• Poll group state. hagroup listgroups reports each member’s status (alive or otherwise), the group’s Needs sync flag, and the current auto-recovery settings. Scrape it on a schedule; alert on any member not alive, on Needs sync: yes persisting, and on HA auto recovery: disabled appearing where it shouldn’t be.
• Enable and ship the HA log. hagroup halog -enable writes failover and recovery events to a client-side log file. Feed it to your SIEM — it is the only record of failover events you will get, and it directly supports ISM event-logging expectations for security-relevant infrastructure.
• Verify replication, don’t infer it. Periodically compare object counts or fingerprints across member partitions. “The group exists” and “the members hold the same keys” are different claims.
• Drill failover on a schedule. An HA group that has never had a member pulled in anger is an untested control. IRAP assessors are entitled to ask for evidence; “it’s configured” is not evidence.

The companion how-to walks through the full build and a proper failover drill step by step. And if your privileged-access layer fronts the same estate, the monitoring philosophy carries over — we take the same evidence-over-assumption line in our CyberArk CorePAS implementation guide.

---

Where this leaves you

You now know what the HA client will and won’t do for you — and that the group is only as real as its last verified synchronisation and its last failover drill. The next problem arrives immediately: HA recovery assumes members can come back without a human present, and on PED-authenticated partitions that means activation policies and unattended restart behaviour, which is its own minefield. That’s the next chapter. The full series lives at The Luna Field Guide.