Multi-region

In short

A hosted SpeculaPro that serves more than one region is built from cells and one directory:

• A cell is a complete SpeculaPro installation for one region: its own PostgreSQL, Kafka, Valkey, optional ClickHouse and application. Cells share nothing. A cell keeps working when every other cell, and the directory, is down.
• The directory is the only global part. It maps a workspace’s slug to its region and a region to its address. It holds no personal data and no customer data.
• A workspace is pinned to one region, and a cell refuses to serve a workspace pinned to another.

A single installation, which is what every self-hosted customer runs, has no region and no directory. Nothing on this page applies to it, and nothing changes in it: with SP_CELL_REGION unset, every check described here passes as it always did.

For the idea in plain words, see Data residency and regions. To run it, see Running more than one region.

The pieces

        browser                         your systems (API keys: spk_gh_…)
           │                                         │
           │ 1. "which region is acme?"              │ always the region's own address
           ▼                                         │
   ┌───────────────────┐                             │
   │     Directory     │◄──── signed calls ────┐     │
   │ slug → region     │      (cells only)     │     │
   │ region → address  │                       │     │
   └─────────┬─────────┘                       │     │
             │ 2. redirect                     │     │
             ▼                                 │     ▼
   ┌──────────────────────────────┐   ┌────────┴─────────────────────┐
   │        Cell: region gh       │   │        Cell: region eu       │
   │ Postgres · Kafka · Valkey    │   │ Postgres · Kafka · Valkey    │
   │ ClickHouse · every role      │   │ ClickHouse · every role      │
   └──────────────────────────────┘   └──────────────────────────────┘
        no connection between cells; the directory never calls a cell

(gh and eu are example region keys; the operator chooses them.)

Piece	Runs	Stores	Depends on
Cell	Every role of the ordinary binary, with SP_CELL_REGION set	Everything about its workspaces	Its own stores only. The directory is needed to create a workspace and to move one, and for nothing else
Directory	The same binary, started as speculapro directory run	Slugs, regions, addresses, a change log, request nonces	Its own small PostgreSQL database. No Kafka, no Valkey, no ClickHouse

What the directory stores

Five tables, in a database of its own with its own migrations. A cell’s database never has these tables, and the directory’s has none of a cell’s.

Table	Holds
dir_regions	Each region’s key, name, public address and status: active, draining (serving, takes no new workspaces) or down (declared unavailable)
dir_workspaces	Each slug, its region, reserved or active, and the region it is being offered to during a move
dir_events	What changed and which cell (or the operator) changed it: slugs and regions only
dir_nonces	The random number of each signed request, until its time window closes
dir_settings	One generated secret: the key that picks the decoy region (see below)

No names of people, no email addresses, no user records. That is why the directory can be hosted anywhere without it being a residency question.

How a person reaches their region

1. The browser opens the global address. The web app is served by the directory there. It first asks GET /api/v1/cell, and the directory answers {"mode": "directory"}, so the page shows Find your workspace instead of the sign-in form.
2. The person types their workspace. The page sends POST /api/v1/directory/resolve.
3. The directory answers with the region’s key, name, address and status.
4. If the status is down, the page says the region is unavailable. Otherwise the browser is sent to <region address>/login?workspace=<slug>, and the person signs in there. Their email and password are never typed at the global address.

The cell’s own sign-in page asks the same GET /api/v1/cell and gets mode: cell with the region’s name, which it shows, and the global address, which it links.

The look-up doesn’t reveal which workspaces exist

A slug that doesn’t exist gets an answer of the same shape: a decoy region, chosen by a keyed hash of the slug, so the same unknown slug always gets the same region. The directory does the same work for a slug that exists and one that doesn’t. The sign-in at the decoy region then simply fails, as it would for a wrong password.

The look-up is limited per client address (20 a minute by default); beyond that the answer is 429 with Retry-After: 60.

How an API call reaches its region

API keys issued by a cell carry the region between the spk marker and the prefix:

spk_gh_k3x9q2m7w4ab_Vn2…Q8
│   │  │            └ secret
│   │  └ 12-character prefix
│   └ region key
└ marks a SpeculaPro key

So a client, or an edge proxy in front of several regions, can route on the key’s shape without asking anyone. Keys from before regions, and every key of a single installation, have no region part (spk_k3x9q2m7w4ab_…) and keep working in their own cell. The two forms can’t be confused: a region key is never twelve characters long.

Integrations should always call their region’s own address. Then they don’t depend on the directory at all.

What a cell answers

Situation	Answer
The key names another region and this cell doesn’t know it	421 Misdirected Request, code wrong_region, with region and, when this cell knows that region’s address, base_url
The key is valid, but its workspace was handed over to another region	The same 421, after the secret is checked
The key is valid, and its workspace is sealed for a move (exported, not yet live in the target)	503 with Retry-After: 60
The key is valid, the workspace is read-only before a move, and the request would change something	503 with Retry-After: 120
Anything else that fails	401 invalid api key, as before

Where a workspace moved to is told only to someone who holds a valid key for it, never to someone guessing key prefixes. The one thing a cell says without checking the secret is what the key itself already says: the region written in it.

See Regions for integrators for the response bodies.

What a cell refuses

A cell reads each workspace’s region pin and move state (cached for five seconds) on every authenticated request, and refuses a workspace it doesn’t serve on every path:

Path	For a workspace pinned elsewhere, sealed or handed over
Sign-in	Refused like a wrong password. A workspace that was handed over is answered with 421 and the new region’s address, so the page can send the person on. The answer is the same whether or not the email has an account, so it doesn’t reveal who did
Sessions	Treated as signed out
API keys	As in the table above
Webhooks	Not delivered from this cell
Notifications (email, SMS, WhatsApp)	Not sent from this cell

This is what makes a misplaced copy harmless: a backup restored into the wrong region, or the old copy of a workspace that moved and hasn’t been purged yet, can’t be signed in to and sends nothing.

How cells and the directory trust each other

Cells call the directory over HTTPS. The directory never calls a cell. Every call is signed with a key that only that cell and the directory hold (32 random bytes, with a version number so it can be rotated).

A request carries five headers:

Header	Content
X-SP-Cell	The calling cell’s region
X-SP-Key	The key’s version
X-SP-Timestamp	Unix seconds
X-SP-Nonce	A random value
X-SP-Signature	v1= and the HMAC-SHA-256 of the method, path and query, region, time, nonce and the body’s SHA-256

The directory:

• checks the signature against that region’s key (any version it currently accepts);
• rejects a timestamp more than five minutes off, and any nonce it has already seen, so a captured request can’t be replayed or pointed at another endpoint;
• lets a cell write only its own region’s entries. A cell can’t release or activate another region’s workspace, and can’t register or change another region’s address;
• moves a workspace between regions only in two halves: the owning cell offers it to a named region, and only that region can accept.

So a cell whose key leaked could damage its own region’s entries in the directory and no one else’s, and could never pull another region’s workspace towards itself.

What a cell asks the directory for

When	Call	If the directory is down
On start, then every five minutes	Register its name and address; read the list of regions (so it can name another region’s address in a 421)	The cell logs a warning, retries every five seconds and keeps serving
speculapro tenant create	Reserve the slug, then mark it active; release it if the create fails	The create fails. Slugs are unique across all regions
Offboarding a workspace	Release its slug	See Offboarding a workspace
speculapro cell sync	Pin unpinned workspaces to this region and list them	Fails; run it again
speculapro workspace move …	Offer, accept or cancel a move	The step fails and can be run again

Signing in at the region’s address, API keys, decisions, ingest, webhooks and everything else in a cell need the directory for nothing.

What depends on what

If this is down	Sign-in at the global address	Sign-in at the region’s own address	API, decisions, ingest, webhooks in a healthy cell	Creating a workspace
The directory	No	Yes	Yes	No
One cell	Other regions: yes	That region: no	Other regions: yes	Other regions: yes

There is no automatic failover between regions, by design: it would move data out of its region. A cell recovers inside its region. See the failover drill.

Moving a workspace

A workspace changes region only through the operator’s move tool: read-only, exported from one snapshot with a row count and digest per table into an encrypted bundle, imported in one transaction that commits only if every table matches, then handed over in the directory, then deleted from the source.

Only Postgres rows travel. The export refuses while the workspace still has unread events in the source’s Kafka or a background run under way, so nothing is written to the source after the snapshot. What a cell derives from the rows is rebuilt, not copied: after the import, the target replays the workspace’s last 192 days of transactions and account events into its own Valkey (counters, velocity windows, first-seen profiles) and copies the workspace into its own analytics store, both before the workspace is served there. The source ends the workspace’s sessions at the handover, and the purge deletes the workspace’s Valkey keys, its rows and its analytics copy. From the start of a move, background jobs begin nothing new for the workspace and its data sources are not read. The cells must hold the same personal-data keys for this to work at all, because encrypted fields travel as ciphertext.

Limits

• Multi-region is for a hosted service. The chart and the settings exist in every build, but a self-hosted installation gains nothing from them.
• No cross-region reads. A person or key that belongs to two workspaces in two regions signs in to each separately. There is no view across regions.
• The directory’s internal API is on the same address as the public look-up. It is protected by the signatures; operators should also restrict /internal/v1 to the cells’ addresses where their ingress or load balancer can.
• No Prometheus metrics specific to cells or the directory yet. The directory serves the standard /metrics and the chart can scrape it; operators also watch its /readyz, its request log and speculapro directory events.

Related

• Data residency and regions
• Running more than one region
• Moving a workspace between regions
• Regions for integrators
• Tenant isolation
• Components