Running more than one region

In short

This page is for people who run SpeculaPro as a hosted service in more than one region. Each region is a complete installation, called a cell. One small global service, the directory, knows which region each workspace is in.

What you will run:

Piece	How many	What it is
Cell	One per region	An ordinary installation (PostgreSQL, Kafka, Valkey, optional ClickHouse, every role) with a region key
Directory	One	The same binary started as `speculapro directory run`, with its own small PostgreSQL database and nothing else

Read Multi-region first for how the pieces fit, and Data residency and regions for what customers are told.

The rules that keep data in its region

Nothing is shared between cells. Not a database, not a Kafka cluster, not a cache, not a bucket. Each cell’s backups stay in its region too: a backup is the data.
A cell serves only workspaces pinned to its region, and refuses any other on sign-in, sessions, API keys, webhooks and notifications.
The directory holds slugs and regions only, so it can be hosted anywhere.
Never change a cell’s region key once it has workspaces. API keys carry it, and every workspace is pinned to it.

Settings

In each cell

Setting	Chart value	Meaning
`SP_CELL_REGION`	`cell.region`	The region key: 2 to 11 characters, a lowercase letter first, then lowercase letters, digits or dashes (`gh`, `eu`, `af-south`). New workspaces are pinned to it and new API keys carry it
`SP_CELL_NAME`	`cell.name`	What people see, for example `Ghana (Accra)`. Default: the region key
`SP_CELL_PUBLIC_URL`	`cell.publicUrl`	Where this cell is reached. The directory sends people and API clients here. Required with a directory
`SP_DIRECTORY_URL`	`directory.url`	The directory’s address, `https` only
`SP_DIRECTORY_KEY`	A key in your Secret	This cell’s signing key, `<version>:<base64 of 32 bytes>`
`SP_DIRECTORY_PUBLIC_URL`	`directory.publicUrl`	The global sign-in address. The cell’s sign-in page links to it (Not your region? Find your workspace). Optional
`SP_DIRECTORY_ALLOW_PRIVATE`	`directory.allowPrivate`	Default `false`. Set `true` when the directory is on your private network (same cluster, a VPN between regions). Link-local and metadata addresses stay refused
`SP_DIRECTORY_TIMEOUT`	`directory.timeout`	Per call. Default `10s`; between `1s` and `1m`
`SP_DIRECTORY_ALLOW_HTTP`	none	Development only. Refused outside development

In the directory

Setting	Chart value	Meaning
`SP_DIRECTORY_DB_URL`	A key in your Secret (`directory.server.dbUrlKey`)	The directory’s own database. Never a cell’s
`SP_DIRECTORY_CELL_KEYS`	A key in your Secret (`directory.server.cellKeysKey`)	The cells’ keys: `gh:1:<base64>,eu:1:<base64>`. Several versions of one region’s key are accepted while it is rotated
`SP_DIRECTORY_ADDR`	`directory.server.port`	Listen address. Default `:8085`
`SP_DIRECTORY_RESOLVE_PER_MINUTE`	`directory.server.resolvePerMinute`	Workspace look-ups one client address may make a minute. Default `20`

The look-up is limited per client address, so the directory must see the real one: set SP_HTTP_TRUST_PROXY and SP_HTTP_PROXY_HOPS to match your load balancer, as for a cell.

What stops start-up

Message	Fix
`SP_CELL_NAME and SP_CELL_PUBLIC_URL need SP_CELL_REGION`	Set the region, or clear both
`SP_DIRECTORY_URL and SP_DIRECTORY_KEY need SP_CELL_REGION (the region this installation serves)`	Set the region
`SP_CELL_REGION must be 2 to 11 characters: …`	Use a valid key
`SP_DIRECTORY_KEY is required with SP_DIRECTORY_URL (speculapro directory keygen gh)`	Generate the key
`SP_CELL_PUBLIC_URL is required with SP_DIRECTORY_URL: …`	Set the cell’s address
`SP_DIRECTORY_URL: …; set SP_DIRECTORY_ALLOW_PRIVATE=true if the directory is on your private network`	Set it, if that is true
`SP_DIRECTORY_ALLOW_HTTP is for development only; the directory must use https`	Use `https`
`directory: SP_CELL_REGION is set; the directory is not a cell and runs with its own configuration`	Don’t give the directory a cell’s configuration
`directory: SP_DIRECTORY_DB_URL is required (the directory's own database, never a cell's)`	Set it

.env.example lists every one of these settings, commented out, under Multi-region.

With Helm

The chart has three modes. split and all are cells (or single installations); directory is the directory and nothing else: no roles, no Kafka, no Valkey.

Value	Meaning
`mode: directory`	This release is the directory. Renders one Deployment (`directory run`), a Service, a pre-install migration Job (`directory migrate up`), an optional Ingress, a PodDisruptionBudget and a NetworkPolicy
`cell.region`, `cell.name`, `cell.publicUrl`	As above, in a cell’s release
`directory.url`, `directory.publicUrl`, `directory.allowPrivate`, `directory.timeout`	The cell’s side
`directory.server.replicas`	Default `2`
`directory.server.migrate`	Default `true`: run the migration Job
`directory.server.ingress.*`	The global sign-in address
`directory.server.httpFrom`	Who may reach the directory’s port. Empty: anyone (the ingress controller)
`directory.server.postgres`	Where its database is: `{ports: [5432, 6432], to: []}`. An empty `to` opens the ports to any address
`networkPolicy.egress.directory`	In a cell: the ports and addresses of the directory, so the cell’s pods may call it

The chart refuses combinations that can’t work: mode: directory with cell.region set; cell.name, cell.publicUrl or directory.url without cell.region; directory.url without cell.publicUrl; a directory.url that isn’t https.

Three example files are in deploy/helm/speculapro/examples/:

File	Use
`directory-values.yaml`	The directory: `mode: directory`, two replicas, its ingress and database
`cell-gh-values.yaml`	A cell, layered on your usual values file: region, addresses, ingress host, egress to the directory
`cell-eu-values.yaml`	A second cell, the same shape

The region keys, names and hosts in them (gh, eu, speculapro.example) are placeholders.

# the directory, once
helm upgrade --install directory deploy/helm/speculapro -n speculapro-directory -f directory-values.yaml

# each cell, next to its usual values
helm upgrade --install speculapro deploy/helm/speculapro -n speculapro -f prod-values.yaml -f cell-gh-values.yaml

In mode: directory the chart renders a ServiceMonitor for the directory’s pods when serviceMonitor.enabled is set, so Prometheus scrapes their /metrics. It renders no alert rules.

How to bring up the directory and a first region

Create the directory’s database and a Secret holding SP_DIRECTORY_DB_URL and an empty SP_DIRECTORY_CELL_KEYS.
Install the directory with mode: directory. The pre-install Job runs speculapro directory migrate up. With no keys yet, it logs SP_DIRECTORY_CELL_KEYS is empty; no cell can register until its key is added.
Generate a key for the region:
Terminal window
```
speculapro directory keygen gh
```
```
# the cell (gh):
SP_DIRECTORY_KEY=1:…
# the directory: add to SP_DIRECTORY_CELL_KEYS (comma-separated)
gh:1:…
```
Put the first line in the cell’s Secret and append the second to the directory’s SP_DIRECTORY_CELL_KEYS. Restart the directory’s pods to load it.
Install or upgrade the cell with its region values. On start it registers its name and address with the directory.
Pin the workspaces it already has:
Terminal window
```
speculapro cell sync
```
Run it with the schema owner’s SP_DB_URL, as for migrate. It pins every workspace that has no region to this one and lists them in the directory. It is safe to run again.

Check both sides:

speculapro cell info            # in the cell
speculapro directory regions    # in the directory

For each further region, repeat steps 3 to 6.

From then on, speculapro tenant create in a cell reserves the slug in the directory first, so a slug is unique across all regions. If the create fails, the reservation is released. Offboarding a workspace releases its slug.

Commands

`speculapro directory`

Run these where SP_DIRECTORY_DB_URL is set (except keygen, which needs nothing).

Command	What it does
`directory run`	Serves the look-up, the global sign-in page and the cells’ signed API
`directory migrate up\|down\|status`	The directory’s own migrations
`directory keygen <region>`	Prints a new version-1 key for a region, as the two lines to paste
`directory regions`	Each region’s key, status, name, address and number of workspaces
`directory region set-status <region> active\|draining\|down`	`draining`: keeps serving, takes no new workspaces. `down`: the global sign-in page says the region is unavailable instead of sending people there. Recorded in the directory’s events
`directory workspaces [--region gh]`	Every slug with its region and state, and moving to … during a move
`directory events [--slug acme]`	The last 200 changes: when, which cell or `operator`, what, which slug

`speculapro cell`

Run these in a cell.

Command	What it does
`cell info`	The region, name and address, and whether the directory is reachable, with the regions it lists. On a single installation: single-cell installation: SP_CELL_REGION is not set, and nothing else needs configuring
`cell sync`	Pins unpinned workspaces to this region and lists every workspace this cell serves in the directory. Reports workspaces pinned to another region (not served here, not listed)
`cell release-slug <slug>`	Frees a slug this region still holds in the directory for a workspace that isn’t in this cell: one whose create failed half-way, or one offboarded while the directory was away. Refused when the workspace exists in this cell

cell sync never takes a workspace from another region. A workspace the directory lists elsewhere is not served by this cell, whatever its database holds.

How to rotate a cell’s key

Generate a new key and give it the next version number: 2:<base64>.
Add gh:2:<base64> to the directory’s SP_DIRECTORY_CELL_KEYS, keeping gh:1:…. Roll the directory.
Set the cell’s SP_DIRECTORY_KEY to the new key. Roll the cell.
Remove gh:1:… from the directory. Roll it.

The cell’s directory key is replaceable: if it is lost, generate a new one and update the directory.

What depends on what

If this is down	Sign-in at the global address	Sign-in at the region’s own address	API, decisions, ingest, webhooks in a healthy cell	Creating a workspace
The directory	No	Yes	Yes	No
One cell	Other regions: yes	That region: no	Other regions: yes	Other regions: yes

Give customers their region’s address as well as the global one. API integrations should always use the region’s base URL: they then have no dependency on the directory.

Monitoring

Monitor each cell as you do a single installation: see Monitoring. In addition:

Watch	Where	Meaning
`cell: the directory could not be reached; will retry`	The cell’s log (WARN)	The cell keeps serving. Only creating and moving workspaces need the directory
`/readyz` on the directory	Its HTTP port and its metrics port	`204` when its database answers
`429` on `/api/v1/directory/resolve`	Your ingress or load balancer logs	A spike is someone probing slugs
`401` on `/internal/v1`	The same	A cell with a wrong or rotated key
`speculapro directory events`	The directory	Who changed which entry

The directory serves /metrics like every pod, and the chart scrapes it (see With Helm). There are no Prometheus metrics specific to cells or the directory yet, and no alert rules for the directory: build your own on its request counts and /readyz.

Failover drill

Run the drill per cell on a schedule you set (the operations note suggests quarterly as a target) and record the result. Recovery targets are a business decision, not a property of the software: SpeculaPro states none. Set them and then measure them.

	Target	Last measured
RPO, cell PostgreSQL (how much data may be lost)	yours to set
RTO, cell (time until the region serves again)	yours to set
RPO, directory	yours to set
RTO, directory	yours to set
Time to detect a cell outage	yours to set

Part A: failover inside the region

A cell’s data must stay in its region, so the first line of recovery is a standby in another availability zone of the same region, never another region.

Announce the drill. Note the time.
Mark the region down, so the global sign-in page says so instead of sending people into an error:
Terminal window
```
speculapro directory region set-status gh down
```
Fail PostgreSQL over to the in-region standby. Note how many seconds of writes were not on the standby: that is the measured RPO.
Check the application reconnected: /readyz on each role, then sign in at the region’s address and open a case.
Check the pipeline: consumer lag for the persist and stream groups returns to zero, the decision API answers, and one test event sent through ingest appears in the app.
Check outbound: send a webhook test from Settings → Webhooks, and check the notification outbox drains.
Mark the region active again. Note the time: that is the measured RTO.
Terminal window
```
speculapro directory region set-status gh active
```
Confirm the other regions saw nothing: their error rates and latencies during the window.

Part B: the directory is lost

Stop the directory’s pods.
Confirm each cell still serves sign-in at its own address, API keys, decisions, ingest and webhooks. Only the global look-up and tenant create fail.
Restore the directory’s database from its backup into a fresh instance and start the pods.
In every cell, run speculapro cell sync. It registers the region again and lists every workspace the cell serves, so the directory is rebuilt from the cells even if the backup was old or empty.
Compare speculapro directory workspaces --region gh with the cell’s own list.

The cells are the source of truth for which workspaces exist. Two things a rebuild can’t know: slugs reserved for creates that were in flight (they are free again), and a workspace move in progress. Finish or abort moves before a planned directory restore; after an unplanned one, check speculapro workspace move status for each open move.

Part C: the region itself is lost

There is no automatic cross-region failover, by design: failing a cell over to another country would move the data out of its region. If a whole region is lost, the choices are:

restore the cell from its backups into the same region, or another site in the same jurisdiction, when it is available again; or
with the affected institutions’ written agreement, and where their regulator allows it, restore into another region as a new cell with the same region key.

Decide in advance which applies per region, and write it into the contract. Institutions record it in their cross-border transfers. Not legal advice: confirm with counsel.

Backup and recovery, per cell

Every cell is backed up on its own, to storage in its own region. See Backups for the ordinary procedure; what multi-region adds:

What	How	Notes
The cell’s directory key	Your secret store	Replaceable
The directory	Its own PostgreSQL backups, anywhere	Small, and rebuilt from the cells with `cell sync`

Restore order for a cell: PostgreSQL to the chosen point in time → secrets and keys → speculapro migrate up → the application → check sign-in and the pipeline → speculapro cell sync → rebuild ClickHouse → mark the region active.