Moving a workspace between regions

In short

A workspace changes region only with speculapro workspace move. It is an operator’s action, done with the customer’s agreement, in eight small steps that each run in the cell they concern and are safe to run again.

The shape of a move:

 source cell                                   target cell
 ───────────                                   ───────────
 start      read-only; offered in the directory
 export     waits until nothing is pending;
            sealed; written to a bundle  ───►  import     loaded and checked; counters and the
                                                          analytics copy rebuilt; not served
                                               verify     compared with the bundle again
                                               flip       live here; the directory points here
 handover   sessions end; answers clients
            with the new region
 purge      the old copy is deleted

There is downtime: the workspace is read-only from start, and unavailable from export until flip. How long depends on the workspace’s size, on how fast you carry the bundle across, and on how much history the target replays to rebuild the workspace’s counters (see What the target rebuilds); SpeculaPro promises no figure.

This page applies only to a multi-region deployment. A single installation has nowhere to move a workspace to; to hand a workspace’s data back and delete it, see Offboarding a workspace.

What is not carried

Read this before you promise a customer anything

A move does not carry everything. It is also printed by export and written into every bundle’s manifest.

• Sign-in sessions. The target starts with none, so everyone signs in again at the new region’s address. The source ends the workspace’s sessions at handover.
• The analytics store (ClickHouse). Not copied. import rebuilds the workspace’s copy in the target from the imported rows, and verify says whether it did. The source’s copy is deleted by the purge.
• Feature state in Valkey (rolling counters, velocity windows, first-seen profiles). Not copied. import rebuilds it in the target by replaying the workspace’s transactions and account events of the last 192 days. With --cold-features, or if the rebuild fails, counters start empty; verify says so, and for how long it shows. Idempotency records and caches are not rebuilt: they expire in the source and start empty in the target. The long-term registry of ingested ids is in Postgres and is carried.
• Events still in the source’s Kafka. Not copied. export refuses while the workspace has events its consumers have not read, or a background run under way, so nothing is written to the source after the export.
• The installation’s sanctions lists. The official lists and the lists an operator imported belong to a cell, not to a workspace. Screening matches against them keep their evidence (list, entry, names, scores, who decided) but lose the link to the source cell’s list version; the target screens against its own copy of the same lists. A workspace’s own imported lists (the list, its versions and its entries) and its cleared false positives are carried.

Before you start

Requirement	Why
Both cells on the same release	The export records the schema version, and the import refuses another: bring both cells to the same release first
The target holds the source’s personal-data keys	See The keys
The rule packs the workspace uses are installed in the target	The import names any that are missing: the workspace uses rule pack … which this cell doesn’t have; install it here first
A database role in the target that may set session_replication_role	A superuser, or GRANT SET ON PARAMETER session_replication_role. Rows are loaded exactly as exported, without triggers
The schema owner’s SP_DB_URL for every step	As for migrate: a move reads and writes every table of the workspace
The cell’s usual environment for export, import, purge and abort	Those steps also use the cell’s Valkey (SP_REDIS_*); export its Kafka (SP_KAFKA_*); and import, purge and abort its ClickHouse when the cell has one (SP_CLICKHOUSE_URL)
Both cells configured with a directory	SP_CELL_REGION, SP_CELL_PUBLIC_URL, SP_DIRECTORY_URL and SP_DIRECTORY_KEY
The directory lists the workspace in the source region	Otherwise: run speculapro cell sync first
Somewhere safe for the bundle at each end, and a safe way to carry it	The bundle is encrypted by default, but its manifest (table names, row counts, the workspace’s slug) is readable. See The bundle
The customer’s written agreement, and a time window they chose	It is their downtime and their data crossing a border

Agree with the customer, in advance:

• Data sources pause by themselves. From start, the workspace’s Kafka topics, file drops and change data capture sources are no longer read: files stay in their folder, topics keep their messages, and a Postgres source’s replication slot holds its write-ahead log on the institution’s database. They are read again in the target after the flip, or in the source after an abort. Nobody has to pause them by hand.
• Network access from the new region. Their Kafka brokers, SFTP servers, buckets, databases (for replication), webhook endpoints and single sign-on will be reached from the target cell’s addresses after the move. Firewalls and allow-lists must admit them.
• Systems that call the decision API need a plan for the window: a 503 is a refused request, not an answer.

A move is a transfer

The workspace’s data leaves one jurisdiction for another. The institution may need its regulator’s agreement or a legal basis under its data protection law, and should record the move in its cross-border transfers. Not legal advice: confirm with counsel.

The keys

Encrypted fields are carried as ciphertext, and look-ups by identifier depend on the blind index. So the target cell must hold:

• every master key version the source has (SP_SECURITY_MASTER_KEYS), and
• the same blind-index key (SP_SECURITY_BLIND_INDEX_KEY).

The bundle carries check values of the source’s keys, never the keys. The import compares them and refuses when the target lacks a master key version or has another blind-index key.

This is a decision to make when you design your regions, not on the day:

• Cells that share these keys can exchange workspaces. A leak of the keys then affects every cell that holds them.
• Cells with separate keys can never exchange workspaces. If two regions must stay strictly apart, give them separate keys.

See Encryption keys.

The bundle

The export writes a directory, the bundle, that holds the whole workspace: transactions, case notes, audit entries, password hashes.

• It is encrypted by default. Each table file is sealed with AES-256-GCM, in chunks, under keys derived from the source cell’s active personal-data master key, a random salt for that export and the file’s name. Files are created readable by their owner only.
• No key is handed over between operators. The target derives the same keys, because it must hold the source’s master keys anyway.
• The manifest stays readable. manifest.json lists table names, row counts, digests and the workspace’s slug, and no rows. manifest.mac proves it is the one the source wrote.
• The import checks the bundle before it loads anything: the manifest, then every file’s SHA-256. A bundle that was changed, cut short or encrypted under a key the target doesn’t hold is refused.

Option	Step	What it does
--plaintext	export	Writes the bundle unencrypted. Only for a case you have thought through, such as inspecting an export on an encrypted volume
--allow-plaintext	import	Accepts a bundle exported with --plaintext. Without it, such a bundle is refused

Encrypted is not disposable

Anyone who holds the cell’s master keys can read the bundle. Keep it where only the operators of the move can reach it, and destroy it after the purge.

What stops during a move

From start, nothing new is begun for the workspace in the source cell:

• reading its data sources (file drops, Kafka topics, change data capture);
• scheduled rules, screening and re-screening;
• cash transaction report drafting;
• the retention purge, scheduled erasures and data protection reminders;
• backtests, risk model trainings and drift checks;
• case deadline and licence notices;
• the clean-up of expired ingest keys, webhook history and data-source history.

Backtests, trainings and screening runs that were queued wait, and are carried to the target as queued. Notifications and webhooks already queued are still sent until export, then stop.

Two things can still write to the workspace after start:

• A run that was already under way: a backtest, a model training, a scheduled rule run, a screening run, a retention purge run, a data source being read or a file being loaded.
• Events still in the source’s Kafka, which the persist and stream roles write when they reach them, and decision events waiting in the decision role’s spool for Kafka.

export looks for both and refuses while either exists. See The export waits for the workspace to drain.

The export waits for the workspace to drain

An export is the workspace as it was at one instant. Anything the source writes to it afterwards would exist only in the source. So export looks first, and refuses while it finds any of:

• Events of the workspace that a consumer has not read yet, on the transaction and account-event topics, for the persist and stream consumer groups (SP_PERSIST_GROUP, SP_STREAM_GROUP and their -events twins). It reads the unread stretch of each partition and counts only this workspace’s events, so a busy cell, whose lag is never zero, doesn’t hold a move up.
• Decision events waiting in the spool for Kafka.
• A background run under way: a backtest, a model training, a scheduled rule run, a screening run, a retention purge run, a data source being read or a file being loaded.

The refusal names what is pending, with counts:

workspace move: the workspace is not drained: acme still has work in this cell that would be written
after the export and not reach eu: 3 events on sp.txn.ingested not yet read by speculapro-stream;
1 backtest under way. It stays read-only and in service here. Wait for the pipeline and the runs to
finish and export again; --wait 10m waits here for them

Refused at this point, the workspace is not sealed: it stays read-only and in service. Run export again later, or let it wait.

Option	What it does
--wait 10m	Waits up to that long for the pending work to finish, looking again every five seconds, then carries on by itself. The default is 0: refuse at once
--skip-pending-check	Exports without looking. For a cell whose Kafka can’t be reached, or a run that will never finish. It is recorded in the audit entry (pending_check_skipped). Whatever is written afterwards shows up as drift at the purge
--drain 45s	The pause after sealing, for requests in flight. The check runs once more after it, because a request in flight at the seal may have queued something

If that second check finds something, the workspace is already sealed. The message says so: export again when the work has finished (--wait waits), or abort the move to return it to service.

A consumer group that has never run counts as having read nothing

The check treats a consumer group with no committed position as starting from the beginning of the topic. In a cell that doesn’t run one of those roles (no stream role, for example), every event of the workspace still on the topic counts as unread, and the export never passes. Such a cell needs --skip-pending-check. Check for running work by hand first.

A run whose worker died doesn’t stay “under way” for long: the worker fails it or queues it again after its stale period, and leases run out.

If the check itself fails (Kafka or Valkey can’t be reached), the export stops with could not tell whether the workspace is drained and the reason.

What the target rebuilds

A cell keeps two things about a workspace outside Postgres. Both are derived from its rows, so neither is in the bundle. The target builds its own from the rows it imported, after the import has committed and before the flip.

Counters and first-seen profiles (Valkey)

1. import deletes any keys the workspace left in the target’s Valkey on an earlier stay.
2. It replays the workspace’s transactions and account events of the last 192 days through the same code the pipeline and backtests use. No rule runs and nothing is written to Postgres.
3. Counters, velocity windows and first-seen profiles then hold what they held in the source.

This happens while the workspace is not served, so nothing else writes its counters. Running import again does not count anything twice: once rebuilt, it is not rebuilt again for that import. The replay is part of the downtime, and takes roughly as long as a backtest over the same history.

Case	What happens
import --cold-features	The replay is skipped, for when a shorter downtime matters more. Counters start empty at the flip
The replay fails	What it wrote is cleared (half a history is worse than none) and the features are recorded as cold. The import itself stands. Run import again before the flip to retry
The command has no Valkey configured	Nothing is rebuilt, and verify says the features start cold

What “cold” means for rules. A rule over a window sees only what arrived since the flip until that window has passed: a rule over a 30-day window sees full history 30 days after the flip. First-seen signals (new device, new counterparty, new country) treat everything as new once. verify names the workspace’s longest window, so you can tell the customer how long it lasts.

The analytics copy (ClickHouse)

import then copies the workspace into the target’s analytics store, as speculapro analytics backfill --workspace acme would.

• If the store can’t be reached, the import still stands. verify says the copy was not rebuilt, and long reads use Postgres meanwhile. Run import again, or the backfill command, to retry.
• A cell without an analytics store skips this.

What the commands print

import, verify (in the target) and flip print one line for each:

velocity features: rebuilt in this cell from 48211 transactions and 1830 account events since 2026-04-04; rules see the history they saw in the source
analytics copy: rebuilt in this cell (48211 transactions, 51907 decisions)

or, cold:

velocity features START COLD in this cell (the operator chose not to rebuild them): counters, velocity windows and first-seen profiles begin empty. …
analytics copy: this cell has no analytics store; everything reads Postgres

The rebuild is in the target’s audit log as workspace.move.rebuild.

How to move a workspace

The examples move acme from gh to eu. Name the person running each step with --operator; it is written to the audit log.

1. Source: start. The workspace becomes read-only, its data sources are no longer read, no new background work begins for it, and the directory records the offer to the target region.

   speculapro workspace move start --slug acme --to eu --operator "Jane Doe"

   move …: acme is read-only in gh and offered to eu.
   Next: speculapro workspace move export --slug acme --dir <dir> --operator … --wait 10m   (it waits for the workspace's events in Kafka and its runs to finish)

2. Source: export. The command first checks that nothing is pending for the workspace, and with --wait waits for it (see The export waits for the workspace to drain). Then the workspace is sealed (not served at all), the command waits 45 seconds for requests in flight and for cached API keys to run out (--drain changes it), checks once more, and writes the workspace from one database snapshot into an empty directory, encrypted (see The bundle).

   speculapro workspace move export --slug acme --dir /secure/acme --operator "Jane Doe" --wait 10m

   It prints the rows and tables exported, the manifest’s SHA-256, and the Not carried list.

3. Carry the bundle (/secure/acme) to the target cell, whole: the table files, manifest.json and manifest.mac.

4. Target: import. The bundle is checked first: the manifest is the source’s, and every file matches it. Then everything is loaded in one transaction, which commits only if every table’s row count and digest in the target equal the manifest. Otherwise nothing is left behind.

   speculapro workspace move import --dir /secure/acme --operator "Kofi Boateng"

   Once the rows are in, the command rebuilds the workspace’s counters in the target’s Valkey and its analytics copy, and prints what it did (see What the target rebuilds). --cold-features skips the counters. The workspace is in the target now, and not served yet.

5. Target: verify (optional, any time). Compares this cell’s copy with the manifest again and lists any table that differs. In the target it also says what was rebuilt, or that velocity features start cold. Read it before the flip: running import again is the way to retry a rebuild that failed, and it only works before the flip.

   speculapro workspace move verify --dir /secure/acme

6. Target: flip. The target accepts the offer in the directory, and the workspace is live there. The global sign-in page now sends its people to the new region. Nobody is signed in yet: people sign in again. The command writes receipt.json into the bundle directory.

   speculapro workspace move flip --dir /secure/acme --operator "Kofi Boateng"

7. Copy receipt.json back to the source’s bundle directory. It proves the target imported exactly this export.

8. Source: handover. Every session of the workspace in the source cell is ended. The cell now answers the workspace’s API keys and sign-ins with the new region and serves nothing else of it.

   speculapro workspace move handover --slug acme --operator "Jane Doe"

9. Source: purge, first without --confirm. It shows how many rows of the workspace are in this cell and anything that changed since the export. Nothing is deleted.

   speculapro workspace move purge --dir /secure/acme --operator "Jane Doe"

10. Source: purge, with the slug typed again. The source’s copy is deleted: the workspace’s keys in Valkey, then its rows in Postgres, then its analytics copy. See What the purge deletes.

    speculapro workspace move purge --dir /secure/acme --operator "Jane Doe" --confirm acme

11. Destroy the bundle directory in both places. It still holds the workspace’s data.

12. With the customer: check that data sources are being read again from the new region, switch integrations to the new region’s base URL and issue new API keys (see below).

What the customer sees

Period	People	API keys and integrations
start to export	Read-only. They can sign in and read; any change is refused with the workspace is read-only while it is moved to another region; try again shortly	Requests that change anything, including ingest and decisions, get 503 with Retry-After: 120. Reads work. Data sources are not read: file drops stay in their folder, and Kafka topics and change data capture streams wait
export to flip	Unavailable. Sign-in fails as for a wrong password	503 with Retry-After: 60: the workspace is being moved to another region; retry shortly
After flip	Sign-in at the global address goes to the new region. Everyone signs in again: the new region starts with no sessions for the workspace	Keys work at the new region’s address. Data sources are read again, from the new region
After handover	Every session of the workspace in the old region is ended (they had stopped working at export). The old region’s sign-in page sends them on to the new one. It answers 421 with the new region for any email, whether or not that email had an account, so it doesn’t reveal who did	The old region answers 421 with the new base URL

Carried: passwords, two-factor enrolment and recovery codes, single sign-on settings, API keys, webhook endpoints, the workspace’s own imported sanctions lists with their versions and entries, its cleared false positives, and everything else in the workspace’s tables, including its audit log.

The AI assistant arrives switched off. Its settings, reviews and usage are carried, but the settings belong to the region they were saved in, so in the target region the AI assistant reads as off, automatic review stops and asking is refused until an administrator switches it on again there, with the approval that takes. The workspace’s own key is carried encrypted and may need entering again. Tell the workspace’s administrators before the move.

Passkeys stop working. A passkey is tied to the host name of the region it was made in. The records are carried, but no device will offer them at the new region’s address. People sign in with their password and authenticator code (or a recovery code), remove the old passkeys under Settings → Sign-in and add new ones. Tell them before the move.

API keys after a move. Existing keys work in the target, but their prefix still names the old region (spk_gh_… in the eu cell). Ask the customer to issue new keys and retire the old ones, so that clients and edge proxies that route on the prefix are right again. See Managing API keys.

How the copy is verified

• Export. Every table with a workspace column, the workspace’s own row, and the workspace’s rows in tables that hold installation rows and workspace rows side by side (the versions and entries of its own screening lists) are read from one repeatable-read snapshot. The manifest records a row count and a digest per table, the schema version, the rule packs in use, the key check values, what was left out and why.
• Import. One transaction. It commits only when every table’s count and digest in the target equal the manifest.
• Purge. It runs only when all three are true: the directory lists the workspace in the target; the target’s receipt names exactly the export being purged; and the source has not changed since that export.

If the source changed after the export

Background jobs leave a workspace in a move alone, and export refuses while a run is under way or events are unread, so this should not happen. It still can when the export was run with --skip-pending-check: a run cut off by the export is carried as running (the target’s worker later fails it or queues it again as interrupted), and whatever is written in the source afterwards is not in the target. The purge then stops and lists each change:

changed since the export: …
workspace move: acme changed in this cell after it was exported: … Those changes are not in the target.

Look at what changed before you decide. --accept-drift deletes those changes with the rest and records in the audit log that it did.

--accept-drift deletes data that exists nowhere else

Use it only when you have read the list and the changes don’t matter (the result of a backtest or a screening run that will repeat in the target). If the list shows transactions or alerts, that data is only in the source: don’t purge. Abort isn’t possible after the flip, so plan the recovery with the customer, for example by sending those records again in the target.

What the purge deletes

In this order:

1. The workspace’s keys in the source’s Valkey: every key that carries the workspace’s id (counters, profiles, idempotency records, decision statistics). If Valkey can’t be reached, the purge stops with nothing was deleted and can be run again.
2. The workspace’s rows in Postgres.
3. Its copy in the source’s analytics store. If ClickHouse can’t be reached at that moment, the deletion is already recorded and the analytics role applies it later; speculapro analytics status shows it until then.

The command prints a line for each, including how many keys were deleted.

How to abort

Before the flip, either side can undo the move:

speculapro workspace move abort --slug acme --operator "Jane Doe"

• In the source, the workspace returns to service: move aborted: acme is back in service in gh.
• In the target, the imported copy is deleted, with the counters and the analytics copy that were rebuilt from it: move aborted: the imported copy of acme was deleted from this cell.

Run it on both sides if the import already happened, and destroy the bundle.

After the flip there is nothing to abort (a finished move is undone by moving it back): move the workspace back with a new move.

How to see where a move stands

speculapro workspace move status --slug acme

It lists every move of that workspace this cell knows: the two regions, which side this cell is, the step reached (started, exported, imported, flipped, handed_over, purged, aborted), who started it and when it last changed. The directory’s view is speculapro directory workspaces (which shows moving to …) and speculapro directory events --slug acme.

What is recorded

Every step is in the audit log of the cell it ran in, as the system, with the operator’s name:

Action	Step	Where it is kept
workspace.move.start	Read-only began	With the workspace: it travels to the target
workspace.move.seal	Sealed before the export	With the workspace
workspace.move.export	Rows, tables, the manifest’s SHA-256, whether the bundle was encrypted, and whether the pending check was skipped	The source cell, at platform level
workspace.move.import	Loaded and verified	The target cell, at platform level
workspace.move.rebuild	What became of the counters and the analytics copy in the target: rebuilt (with the counts), cold or failed (with the reason)	The target cell, at platform level
workspace.move.flip	Live in the target	With the workspace
workspace.move.handover	The source points clients on; how many sessions were ended	The source cell, at platform level
workspace.move.purge	The source’s copy deleted, whether drift was accepted, and how many Valkey keys were deleted	The source cell, at platform level
workspace.move.abort	Undone	With the workspace in the source; at platform level in the target

Platform-level entries name the workspace as their subject and stay in that cell after the workspace has gone, like workspace.offboard. The directory keeps its own events for the offer, the acceptance and any cancellation.

If a step is refused

Message	Meaning
--to must be another region (this cell is gh)	Name the target region
acme is already in a move (…)	See workspace move status
the workspace is not drained: acme still has work in this cell …	Events are unread or a run is under way. See The export waits for the workspace to drain
could not tell whether the workspace is drained: …	The cell’s Kafka or Valkey couldn’t be asked. Fix the connection, or see --skip-pending-check
purge: the workspace's keys in this cell's feature store could not be deleted, so nothing was deleted	Valkey couldn’t be reached. Run the purge again
this export is for region eu; this cell is …	The bundle was carried to the wrong cell
this cell's blind-index key … is not the source cell's / doesn't hold the source cell's PII master key version N	See The keys
import needs a role that may set session_replication_role	Use the right database role
the analytics store is still deleting an earlier copy of this workspace	The workspace was here before; wait for the analytics role to finish deleting, then import
this bundle is not encrypted. Export it again without --plaintext, …	The bundle was exported with --plaintext. Export again, or pass --allow-plaintext if that was intended
the bundle is encrypted under PII master key version N, which this cell doesn't hold	See The keys
the bundle has no manifest.mac / manifest.json is not the one the source cell wrote	The bundle is incomplete or its manifest was changed. Carry it again, whole
… doesn't match the manifest (SHA-256 …): the bundle was changed or damaged / can't be decrypted: the bundle was changed or damaged, or this cell's key is not the one it was encrypted with	A table file was changed, cut short or damaged on the way. Carry the bundle again
the imported rows don't match the export: …	Nothing was left behind. Check the bundle arrived whole and import again
the directory holds no offer of acme to region eu (was the move aborted in the source?)	The source cancelled the offer
the directory still lists acme in region gh: the target hasn't flipped yet	Flip before the handover
… has no receipt.json: copy it back from the target after the flip	Step 7
the receipt is for another export; the target must import and flip this cell's latest export	The source was exported again after the bundle was taken

Limits

• One workspace at a time, by hand. There is no bulk move and no scheduling.
• The length of the downtime is yours to measure. It includes the replay that rebuilds the counters. Rehearse with a copy of similar size.
• A rebuild can only be retried before the flip. After it, counters that started cold refill from new traffic only, and a missing analytics copy is rebuilt with speculapro analytics backfill --workspace <slug>.
• Decisions are refused during the window. The decision API has no read-only answer.
• Institution-side replication slots stay. From start, a change data capture source is not read, but its replication slot stays on the institution’s database and holds write-ahead log until the source is read again in the target. Its lag is still watched until export. Keep the window short, or agree with their DBA beforehand. See Postgres replication for DBAs.

Related

• Running more than one region
• Multi-region
• Data residency and regions
• Encryption keys
• Analytics store
• Offboarding a workspace