Running roles

In short

The speculapro program can do several jobs, called roles. You choose which ones each running copy does. A small installation runs every role in one process. A large one runs each role on its own group of machines, so that, say, a month-end surge of incoming data doesn’t slow down the analysts’ web app.

Role	Job	Opens an HTTP port?	Database connection
api	Web app, sign-in, management API	Yes (SP_HTTP_ADDR)	SP_DB_URL (speculapro_app)
ingest	The ingest API: receives transactions, customers, accounts, account events and FX rates	Yes (SP_HTTP_ADDR)	SP_DB_URL (speculapro_app)
connector	Reads institution Kafka topics and SFTP / S3 file drops set up under Settings → Data sources, through the same ingest service. Needs outbound access and local disk. See Connector	No	SP_DB_URL and SP_DB_SYSTEM_URL
decision	The real-time decision API: answers allow, step-up, review or block before a payment goes through, and takes outcome feedback. See The decision role	Yes (SP_HTTP_ADDR)	SP_DB_URL and SP_DB_SYSTEM_URL
persist	Copies transactions, decisions and account events from Kafka into Postgres, and queues decision webhooks. See Persister	No	SP_DB_SYSTEM_URL (speculapro_system)
stream	Keeps the running totals rules need, evaluates enabled rules on every transaction, records hits and raises alerts. See Stream evaluator	No	SP_DB_SYSTEM_URL (speculapro_system)
worker	Housekeeping (partitions, old idempotency keys, old webhook history), sending notifications and delivering webhooks. See Worker	No	SP_DB_SYSTEM_URL (speculapro_system)
analytics	Optional. Copies transactions and decisions from Kafka, and rule hits, alerts and cases from Postgres, into the analytics store (ClickHouse). Only where SP_CLICKHOUSE_URL is set	No	SP_DB_SYSTEM_URL (speculapro_system)

all means the seven standard roles: api, decision, ingest, connector, persist, stream and worker. When SP_CLICKHOUSE_URL is set, all also runs analytics; you don’t name it. A process asked to run analytics without that setting stops at start-up with the analytics role needs SP_CLICKHOUSE_URL.

Every process, whatever its roles, also opens the metrics listener on SP_METRICS_ADDR (:9090 by default).

The global directory of a multi-region deployment is not a role. It is started with speculapro directory run, with its own configuration and database, and SP_ROLES doesn’t apply to it.

How to choose roles

Use the --role flag, or set SP_ROLES in the environment. The flag wins if both are set. Separate several roles with commas and no spaces.

speculapro run                          # SP_ROLES, default "all"
speculapro run --role=all
speculapro run --role=api
speculapro run --role=ingest,persist

When the process starts, it logs the roles it is running:

level=INFO msg="speculapro started" roles="[ingest persist]" version=1.0.0 env=production

api, ingest and decision share one port

If a process runs more than one of api, ingest and decision, they are served on the same address (SP_HTTP_ADDR). A process with only ingest serves the ingest API and the health endpoints: no web app, no sign-in, no API-key management. A process with only decision serves POST /api/v1/decisions, POST /api/v1/feedback and the health endpoints. A process that doesn’t run a role doesn’t serve that role’s endpoints.

When you split them, route by path:

Method and path	Send to
POST /api/v1/decisions (exactly this path) and POST /api/v1/feedback	decision nodes
/api/v1/ingest/	ingest nodes
Everything else, including GET /api/v1/decisions, GET /api/v1/decisions/stats, GET /api/v1/decisions/{id} and the other decision log pages people read, and POST /api/v1/decisions/{id}/label (an analyst labelling an outcome)	api nodes

Route on the method as well as the path: /api/v1/decisions is served by decision nodes for POST but by api nodes for GET, and a rule that sends everything under /api/v1/decisions to decision nodes breaks the Decisions page.

Two processes on one machine need two metrics ports

Every process opens SP_METRICS_ADDR. If you run two SpeculaPro processes on one machine, give each its own port (for example :9090 and :9091), or the second one stops at start-up because the port is taken.

Every role needs the same basic settings

Whatever the roles, every process connects to PostgreSQL, Kafka and Redis/Valkey at start-up and loads the encryption keys. So each process needs the full set of SP_DB_*, SP_KAFKA_*, SP_REDIS_* and SP_SECURITY_* settings, even a process that only runs worker. Keep one shared configuration and change only SP_ROLES (and SP_PERSIST_WORKERS / SP_STREAM_WORKERS) per machine.

decision, persist, stream, worker and analytics need SP_DB_SYSTEM_URL in production

Without it they fall back to SP_DB_URL (speculapro_app), where row-level security would block their cross-workspace reads and writes. A process running any of these roles checks this at start-up and refuses to start with persist/stream/worker/decision roles and the provider console: database role cannot bypass row-level security (set SP_DB_SYSTEM_URL to the speculapro_system role). Every process needs the setting where the database roles are separate: api and ingest read the licence through it. On a hosted installation with the provider console on, the api role also serves the console on its own listener (SP_PROVIDER_ADDR): see Setting up the provider console. See Database roles.

How many of each

Role	How many	What limits it
api	2 or more behind a load balancer, for availability	Analyst traffic is light; add more for resilience, not speed
ingest	2 or more behind a load balancer	Each process is stateless. Add processes until Kafka or PostgreSQL becomes the limit
connector	1; 2 for resilience	File drops are leased and Kafka is a consumer group, so extra connectors share the work safely. Needs local disk for the largest file × SP_CONNECTOR_CONCURRENCY
decision	2 or more behind a load balancer, close to Redis/Valkey	Each process is stateless. Redis/Valkey is usually the limit, not the process; see The decision role
persist	Enough workers in total to keep up, up to the partition count	Kafka gives each partition to exactly one worker. With 64 partitions, at most 64 workers do anything; the rest wait idle
stream	Enough evaluators in total to keep up, up to the partition count	Same rule as persist, with its own consumer group. Evaluating rules costs more per transaction than writing it, so expect to need more stream evaluators than persist writers
worker	1 is enough; 2 for resilience	Only one does the work at a time (see below)
analytics	1 is enough; 2 for resilience. None without the analytics store	Its Kafka consumers (SP_CLICKHOUSE_WORKERS per process, default 2) follow the same partition rule as persist, with their own consumer group. The copy from Postgres takes a lock, so only one process does it at a time. See Analytics store: how many

Sizing the persister

Each persist process runs SP_PERSIST_WORKERS parallel writers (default 4). The total across all persist processes should not exceed the partition count of txn.ingested (64 by default):

Persist processes	SP_PERSIST_WORKERS	Total writers	With 64 partitions
1	4	4	Fine for a small installation
2	8	16	Fine
4	16	64	The maximum that does useful work
8	16	128	64 writers sit idle. Wasted connections

Every writer holds database connections, so check the totals against PostgreSQL’s limits. See Connections.

Sizing the stream evaluator

Each stream process runs SP_STREAM_WORKERS parallel evaluators (default 4), and the same limit applies: the total across all stream processes should not exceed the partition count of txn.ingested. Each evaluator uses database and Redis connections. See Stream evaluator: sizing.

Running more than one worker is safe

Each maintenance run takes a PostgreSQL advisory lock first. If another worker already holds it, the run is skipped. Two or three worker processes therefore never do the same work twice; the extras simply take over if one stops.

Webhook deliveries are shared the same way. Each worker claims a batch of up to 200 due deliveries and reserves them for as long as the whole batch can take: the batch is sent SP_WEBHOOKS_CONCURRENCY at a time, each request allowed SP_WEBHOOKS_TIMEOUT plus 5 seconds to record it, with 30 seconds to spare. With the defaults (16 at a time, 10 seconds) a full batch is reserved for about 3 minutes 45 seconds. A reservation never exceeds 10 minutes: with slow timeouts and low concurrency, a worker claims fewer deliveries, down to one round of SP_WEBHOOKS_CONCURRENCY, so a worker that crashes can’t hold deliveries up for longer than that. Two workers therefore never send the same attempt, however slow the receivers are. If a worker stops mid-batch, its unfinished deliveries are sent by another worker once the reservation runs out; one that was in flight at the moment it stopped may then arrive twice. That is one reason receivers de-duplicate on webhook-id.

Notifications use the same idea with a fixed 2-minute reservation: a worker sends a claimed batch one message at a time, starts a message only if it can finish within the reservation, and hands the rest back for the next run.

The decision role

The decision role answers real-time decisions: your payment system asks should this payment go through? and gets allow, step_up, review or block back, within the time budget the workspace has set. What happens on each request:

1. The answer is computed from Redis/Valkey (running totals and recent activity), the workspace’s rules and lists (held in memory, re-read every SP_DECISION_STATE_TTL) and cached customer profiles (SP_DECISION_PROFILE_TTL). Postgres is read only when a profile isn’t cached. Where screening is on, the process also holds the screening list index in memory, to screen a counterparty with no stored result: see Inline screening in the decision role.
2. If the checks don’t finish within the workspace’s budget, or a dependency fails, or the process already has SP_DECISION_MAX_INFLIGHT decisions in progress, the workspace’s failure policy answers instead and the decision is marked degraded.
3. The answer goes back to the caller before anything is written. The decided transaction is then handed to Kafka, from where the persister stores it (with its decision) and the stream evaluator monitors it like any other transaction.

What the role keeps separate

A decision must never wait behind other work, so the role keeps its own resources:

Resource	Why
Its own Redis connection pool, of SP_REDIS_POOL_SIZE	Rate limits, ingest and the other roles in the same process can’t take the connections a live decision is waiting for
Its own Kafka producer, with a large buffer	Handing a decided transaction to Kafka never delays the answer
A spool in Redis	If Kafka can’t take a decided transaction, it is parked in Redis and replayed when Kafka is back. See Kafka: when Kafka is down

Sizing it

• Redis/Valkey is the limit, not the decision process. A decision reads and updates many running totals, and each rule window reads several time buckets. On Cluster, the keys of each workspace and customer, account, counterparty or device are grouped together, so the load spreads across shards and capacity grows with them. See Redis / Valkey: real-time decisions.
• Raise SP_REDIS_POOL_SIZE on busy decision nodes: a decision that has to wait for a free connection may run out of budget.
• Watch the degraded rate (speculapro_decision_degraded_total, by cause). A rising timeout or overload count means you need more capacity; dependency means something failed. See Monitoring.
• Measure with your own rules and hardware before go-live. See Load testing.

Health checks for decision nodes

Check decision nodes with /readyz?role=decision, both for readiness probes and for load balancer health checks. It checks PostgreSQL only.

A decision node still answers when Redis or Kafka can’t be reached, with the workspace’s failure policy, and parks decided transactions in the spool. A check that included Kafka or Redis would fail on every decision node at the same moment and remove them together, and your payment system would get connection errors instead of answers: fail-open would never happen.

Give each process the roles it serves: /readyz?role=api for the web app (PostgreSQL), /readyz?role=ingest (PostgreSQL and Kafka), /readyz?role=api,ingest for both. A process running every role uses /readyz?role=all, which checks PostgreSQL only, so decisions keep failing open there too. A bare /readyz is safe too: on any process that answers decisions (a decision-only node, or one that also runs api and ingest, including all) it checks PostgreSQL only, like ?role=all, so fail-open holds behind any load balancer. On a process without the decision role it checks what that process’s roles need, combined: ingest alone checks PostgreSQL and Kafka. The Helm chart and the production Compose file already choose the right check for each process. See Health checks and Kubernetes with Helm: health probes.

Example layouts

Small: one or two machines

# on each machine
SP_ROLES=all
SP_PERSIST_WORKERS=4
SP_STREAM_WORKERS=4

Both machines run everything. Put a load balancer in front for the web app and the ingest API. With two machines there are 8 persist writers, 8 stream evaluators and 2 workers (only one active at a time).

Large: a group per role

Group	SP_ROLES	Machines	Notes
Web	api	2	Behind the analysts’ load balancer
Ingest	ingest	3+	Behind the integration load balancer, /api/v1/ingest/
Decision	decision	3+	Behind the payment load balancer, POST /api/v1/decisions and /api/v1/feedback; place close to Redis/Valkey; raise SP_REDIS_POOL_SIZE
Persist	persist	4	SP_PERSIST_WORKERS=16: 64 writers for 64 partitions
Stream	stream	4	SP_STREAM_WORKERS=16: 64 evaluators for 64 partitions; place close to Redis/Valkey
Maintenance	worker	2	One active, one standby
Analytics (optional)	analytics	1–2	Only with the analytics store. Needs network access to ClickHouse, as do the Web and Maintenance groups, which read it

Scale each group on its own measurements: ingest by request latency and error rate, decision by its degraded rate and latency, persist and stream by their consumer lag.

What happens on shutdown

On SIGTERM (for example systemctl stop, docker stop or a Kubernetes rollout), SpeculaPro stops accepting new HTTP requests and gives the ones in flight up to SP_HTTP_SHUTDOWN_TIMEOUT (20 seconds by default) to finish. Persist writers and stream evaluators stop after their current batch; anything not yet committed is picked up by another one, without duplicates.

A decision process then finishes its own work in order, so no decided transaction is lost on a clean stop:

1. It gives Kafka up to 10 seconds to take every decided transaction still buffered. Any it couldn’t deliver by then are parked in the Redis spool, to be replayed by another decision node or after the restart.
2. It writes the replay records still queued (up to 5 seconds), so a payment system that retries gets the original answer.
3. Only then does it close its Redis connection.

Give a decision node enough time to stop: at least SP_HTTP_SHUTDOWN_TIMEOUT plus 15 seconds, so 35 seconds with the defaults. Kubernetes’ default terminationGracePeriodSeconds of 30 seconds is too short in the worst case. The Helm chart sets 60 seconds, after a 10-second preStop pause, and refuses anything below the pause plus 45; the production Compose file sets stop_grace_period: 60s. With those, rolling restarts are safe.

Related

• Persister
• Stream evaluator
• Worker
• Analytics store (ClickHouse)
• Configuration reference
• Load testing
• Architecture: one program, several roles