How to build and failover-test a Luna HSM HA group

This is the hands-on companion to Luna HSM HA groups: what actually replicates — and what silently doesn’t. That chapter covers the architecture and the failure modes; this one is the build: two partitions on two Luna Network HSM 7 appliances, one HA group, auto-recovery configured, and — the part most teams skip — an actual failover drill with evidence at the end. Commands are verified against the Thales Luna 7.x documentation; where behaviour varies by client or firmware version, we say so inline. All of this runs in LunaCM on the client, because Luna HA is a client-side construct.

Step 1 — Verify the prerequisites

Do not skip this step; almost every HA build failure traces back here. You need:

• Two partitions on different HSMs, same cloning domain. Cloning is the replication mechanism, so members must share a domain. Before firmware 7.8.0 that means initialised with the identical domain secret; from 7.8.0 with client 10.5.0+, Extended Domain Management only requires one domain in common with the primary. If the domains don’t match, there is no fix short of re-initialising a partition — see the cloning domains chapter for why.
• Matching authentication and policies. Same authentication method (both password-auth or both PED-auth), HSM policies 7 (cloning) and 16 (network replication) on, partition policies 0 and 4 on. Same firmware family, appliances time-synced via NTP.
• Same Crypto Officer credential on every member. The client authenticates to all members with one credential; set the CO password or challenge secret identically before you start.
• Both partitions registered and visible to this client. NTLS or STC established to both appliances, client assigned to both partitions.

Confirm visibility before going further:

lunacm:> slot list

   Slot Id ->              0
   Label ->                par-prod-a
   Serial Number ->        154438865288
   ...
   Slot Id ->              1
   Label ->                par-prod-b
   Serial Number ->        1238700701515

Note both serial numbers — you’ll use them throughout. Serial numbers are stable; slot numbers can shuffle, which is why the following steps prefer -serialnumber over -slot.

Step 2 — Create the group on the primary partition

lunacm:> hagroup creategroup -label prodHA -serialnumber 154438865288

        Enter the password: ********

        New group with label "prodHA" created with group number 1154438865288.
        Group configuration is:

                HA Group Label:  prodHA
               HA Group Number:  1154438865288
              HA Group Slot ID:  Not Available

The partition you name here becomes the primary member — new keys are always generated on the primary, then replicated. The password prompted for is the Crypto Officer credential. LunaCM may restart after group creation and the virtual slot appears in the slot list; expect slot numbering to change.

Step 3 — Add the second member

lunacm:> hagroup addmember -group prodHA -serialnumber 1238700701515

        Enter the password: ********

        Member 1238700701515 successfully added to group prodHA.

If the partition being added already contains objects, LunaCM asks whether to copy them into the group or remove them. Answer deliberately: copy merges its objects into every member; remove deletes them. In a fresh build, the partition should be empty and the question shouldn’t arise. Repeat this step for any additional members, and use hagroup addstandby -group prodHA -serialnumber <serial> if a member (a DR-site HSM, typically) should replicate everything but only serve traffic when all active members are down.

Step 4 — Verify synchronisation

lunacm:> hagroup listgroups

        HA auto recovery:  disabled
        HA recovery mode:  activeBasic
        Maximum auto recovery retry:  0
        Auto recovery poll interval:  60 seconds
        HA logging:  disabled
        Only Show HA Slots:  no

                HA Group Label:  prodHA
               HA Group Number:  1154438865288
              HA Group Slot ID:  5
                Synchronization:  enabled
                  Group Members:  154438865288, 1238700701515
                     Needs sync:  no

        Slot #    Member S/N                 Member Label    Status
        ======    ==========                 ============    ======
             0    154438865288                 par-prod-a     alive
             1    1238700701515                par-prod-b     alive

Three things to check: every member shows alive, Needs sync is no, and — note it now — auto recovery is disabled and HA logging is disabled. Those are the defaults, and both are wrong for production. If Needs sync shows yes, force a synchronisation:

lunacm:> hagroup synchronize -group prodHA

One version-dependent trap: on client 10.4.0+, synchronising a group that mixes FIPS and non-FIPS partitions will fail to replicate non-FIPS-approved keys to the FIPS members — LunaCM returns an error for those keys and synchronises the rest. Don’t mix FIPS postures within a group.

Step 5 — Configure auto-recovery

Auto-recovery is off by default: a failed member stays out of the group until a human intervenes. Setting a retry count is what turns it on:

lunacm:> hagroup retry -count -1
        HA Auto Recovery Count has been set to -1

lunacm:> hagroup interval -interval 120
        HA Auto Recovery Interval has been set to 120 seconds.

-count -1 retries forever; a positive value (1–500) sets a finite budget of attempts, spaced by the interval (60–1200 seconds, default 60). Prefer -1 unless you have a specific reason not to: a finite count that expires during a long outage leaves the member permanently stale until someone runs hagroup recover -group prodHA manually. Then set the recovery mode — activeBasic runs a background thread that detects, recovers, and resynchronises returning members; activeEnhanced additionally restores sessions and login state after a total group outage (mode names and availability are client-version-dependent; check hagroup listgroups output for what your client reports).

Critically: these settings live in each client’s Chrystoki.conf / crystoki.ini. Configure them on every application server that uses the group, not just the admin workstation.

Step 6 — Enable HA-only mode and logging

lunacm:> hagroup haonly -enable
        "HA Only" has been enabled.

lunacm:> hagroup halog -enable
        HA Log was successfully enabled.

lunacm:> hagroup halog -show
        HA Log:  enabled
        Log File:  C:\Program Files\SafeNet\LunaClient\halog\haErrorLog.txt
        Max File Length:  500000 bytes

HA-only mode hides the physical member slots from applications, leaving only the virtual slot. Without it, an application (or an operator’s script) can write to a single physical member directly — creating keys that exist on one HSM only, and whose direct changes the next synchronisation can overturn. Enable it unless you have a documented reason not to. The HA log records failover and recovery events on the client (default path shown is the Windows client; Linux clients log under the LunaClient install path) — set -path and -maxlength to suit, and ship the file to your SIEM.

Step 7 — The failover drill

The group is now built correctly. It is still an untested control. Run this drill in a maintenance window before go-live, and on a recurring schedule after — it’s the evidence an IRAP assessment will want, and the ISM’s expectations on testing continuity controls apply squarely here.

• 1. Start load. Run your application, or a test harness (Thales’s multitoken tool works) generating continuous signing operations against the virtual slot.
• 2. Pull a member. Disconnect the network path to one appliance (pull the cable, down the switch port, or firewall the NTLS port). Don’t be gentle — a real failure won’t be.
• 3. Observe the application. Expect a pause of up to roughly 20 seconds (the client’s connectivity/command timeout, tunable via ProbeTimeout in the client configuration file) before pending operations reschedule onto the surviving member. Any operation mid-flight on the failed member may return an error — confirm your application retries rather than crashes. This is the single most valuable observation in the whole drill.
• 4. Confirm the client saw it. hagroup listgroups should show the pulled member no longer alive; the HA log should have a corresponding entry.
• 5. Create a key during the outage. Generate a test key in the virtual slot while the member is down. This is the replication-catch-up test.
• 6. Restore and watch recovery. Reconnect the member. With auto-recovery configured, the recovery thread should return it to alive within your retry interval and synchronise it — including the key created in step 5. If auto-recovery is off or the retry budget expired, recover manually: hagroup recover -group prodHA, then hagroup synchronize -group prodHA.
• 7. Verify, don’t assume. Confirm Needs sync: no, both members alive, and the step-5 test key present on the recovered member. Then delete the test key and record the drill: date, member pulled, observed failover time, application behaviour, recovery time.

A note for PED-authenticated partitions: recovery of a rebooted appliance depends on partition activation state — a member that comes back needing a PED presentation will not rejoin unattended. That interaction is the subject of the next chapter; drill it explicitly if you run PED auth.

Step 8 — Monitoring checklist

What to poll and what to alert on, once the group is in service:

Check	Source	Alert when
Member status	hagroup listgroups (scripted, e.g. every 5 minutes)	Any member not alive
Sync state	hagroup listgroups	Needs sync: yes persisting beyond one poll cycle
Auto-recovery config	hagroup listgroups header	HA auto recovery: disabled or retry count 0 on any production client
Failover / recovery events	HA log file (shipped to SIEM)	Any failover event; recovery attempts exceeding a threshold
Replication integrity	Object count / fingerprint comparison across member partitions	Counts diverge between members
Failover drill	Change calendar	More than 6–12 months since last evidenced drill

Remember the monitoring is per-client: a healthy group as seen from the admin box says nothing about the application server whose auto-recovery was never enabled. Poll from the machines that matter.

---

Where this leaves you

You have a built, synchronised, auto-recovering HA group — and, more importantly, drill evidence that it fails over the way you think it does. The next problem is unattended operation: everything above assumed a client that can reconnect on its own, and on PED-authenticated partitions that depends on activation policies surviving reboots without a human holding a PED key. That’s where the series goes next — the full chapter list is at The Luna Field Guide.