Key rotation drill

In short

Rotating the master key means adding a new key that encrypts all new personal data from now on, while keeping the old key so existing data stays readable. The mechanism is described in Encryption keys. This page turns it into a drill: a procedure you rehearse in a test environment, then carry out in production with a checklist, so it goes smoothly when you need it in a hurry.

What rotation does and doesn’t do today

Rotation protects new data with the new key. Existing data stays encrypted with the old key, because SpeculaPro doesn’t yet have a command to re-encrypt it. So:

• you must keep every old key version in SP_SECURITY_MASTER_KEYS, and
• after a suspected leak, rotation limits further exposure but doesn’t protect data already encrypted with the leaked key. Treat that as a security incident under your institution’s procedures.

The blind index key never rotates. See why.

When to rotate

Situation	Rotate
Routine, on the schedule in your security policy (yearly is common)	Planned, in a maintenance window
A person who had access to the keys leaves	Soon, planned
A key may have been exposed (pasted in a ticket, on a lost laptop, in a leaked backup with its keys)	Immediately, and open a security incident

Roles in the drill

Two people, so no single person handles the whole key lifecycle alone:

Role	Does
Operator	Generates the key, updates the secret store, restarts services, verifies
Key custodian	Witnesses, stores the new key’s offline backups, signs the record

Before you start

• You know where the current value of SP_SECURITY_MASTER_KEYS lives (your secret manager) and can update it.
• You know every place a SpeculaPro process runs: every server, container or pod.
• You have a current database backup (see Backups).
• The key custodian has the materials for offline backups ready (for example, two encrypted USB drives or sealed envelopes).
• You have rehearsed this procedure in a test environment within the last year.

How to rotate

1. Write down the current key versions. Look at the current value and note the versions, for example 1. The new key takes the next unused number, here 2. Versions can’t be reused, so check your key record, not just the current value.

2. Generate the new key, on a trusted machine:

   speculapro keys generate

   Take only the base64 part of the SP_SECURITY_MASTER_KEYS=1:… line. Ignore the version number it prints and the blind index key line: you’re keeping your existing blind index key.

3. Build the new value, with the new key first and every old entry kept after it:

   # before
   SP_SECURITY_MASTER_KEYS=1:OLDKEY…=
   # after
   SP_SECURITY_MASTER_KEYS=2:NEWKEY…=,1:OLDKEY…=

4. Back up the new key offline before you deploy it. The custodian makes at least two offline copies, in separate locations, and records the version number and date. A key that reaches production before it is backed up is a key you can lose.

5. Update the secret store with the new value.

6. Restart every SpeculaPro process, one at a time if you need to stay available. Every process must pick up the new value. Until they all have, some will still encrypt with the old key. That’s harmless but delays the rotation.

7. Verify:

   • Each process starts without key errors. A malformed value stops the process at start-up with a clear message. See Error messages.
   • The health check reports ok. See Monitoring.
   • A user who may view personal data can open an existing customer record and see their details. This proves old data is still readable.

8. Record the rotation: date, old and new versions, who did it, who witnessed it, where the backups are, and the reason (routine or incident). Keep this with your security records.

Never remove the old entry

Removing 1:… makes every value encrypted under version 1 unreadable, with errors such as master key v1 not configured. If you see that error after a rotation, put the old entry back at once.

If something goes wrong

Symptom	Cause	Fix
Process won’t start: master key vN must be 32 bytes, base64	The new key was copied incompletely	Copy it again from the generated output, or generate a new one (and record the version)
Process won’t start: duplicate master key version N	The new key reused an existing version number	Use the next unused number
Existing personal data can’t be read: master key v1 not configured	An old entry was dropped	Restore the previous entries after the new one and restart
Some new records were encrypted with the old key	Not every process was restarted	Restart the stragglers. No harm done; those records stay readable

Rehearsing the drill

Run the whole procedure once a year in a test environment that mirrors production, ideally the one you use for the restore drill. Time it, note anything unclear, and update your internal runbook.

Related

• Encryption keys
• Data protection
• Backups and restore
• Production checklist