Worker

In short

The worker (role worker) is SpeculaPro’s caretaker. It runs in the background, does a little housekeeping on a timer, and makes sure the database is ready for next month before next month arrives. Its jobs are listed below.

Job	What it does	Why it matters
Partitions	Creates the monthly partitions for transactions, decisions, rule_hits, audit_log and customer_events, from one month back to three months ahead, matching the range the migrations create. The worker creates them through the owner function sp_ensure_monthly_partitions, which grants.sql lets the system role call	Without them, new rows fall into a catch-all default partition, which slows queries and complicates archiving
Idempotency keys	Deletes the record of each external_id you sent once it is older than SP_WORKER_INGEST_KEY_RETENTION (90 days by default)	Keeps the duplicate-protection table from growing forever
WhatsApp code counters	In the same housekeeping run, deletes the installation-wide count of WhatsApp verification codes per number (whatsapp_code_numbers) once an entry is older than 2 days. Logs worker: expired WhatsApp code counters with the count	The per-number limit only looks back one day; the counter holds keyed hashes of numbers, not the numbers, and is kept no longer than needed
Notifications	Sends the email and text messages waiting in the outbox, and every minute warns about cases due within the hour (missed deadlines are the Case deadlines job below)	Without it, nobody is told about assignments, approvals or deadlines. See Notifications
Licence (self-hosted)	Every hour: records the latest time seen, counts each workspace’s transactions per day for the usage report, and notifies administrators when the licence needs attention. With SP_LICENSE_CHECKIN_URL set, also posts a usage report once a day	Keeps the usage counts and reminders current. The licence state itself is worked out from the current time whenever it is read, so it is right even while no worker runs; only the usage counts and the latest-time-seen record wait for the worker. It never affects monitoring. See Licensing
Data protection	After each housekeeping run: the retention purge for workspaces that switched it on (a new run once the last is 20 hours old, in batches of 5,000 rows, resuming after a stop), scheduled erasures that have come due, the repeated check of open data subject requests against cases and reports (the MLRO is notified when one becomes flagged), and reminders for data subject requests due within three days and breach notifications due within 24 hours. Separately, about once a minute, it applies what the purge and erasures removed to the optional analytics store, retrying until ClickHouse accepts it and applying each one a second time about 10 minutes after it was recorded	Without it, nothing past its retention period is removed, postponed erasures never happen, and nobody is reminded of legal deadlines
Backtests	Runs the backtests people start on a rule’s page or in the rule builder	Without it, backtests stay Waiting for a worker. See Backtests
Risk models	Runs the risk model trainings people start on the Models page or you queue from the command line, checks workspaces’ retraining schedules once a minute and queues the trainings that are due, measures drift for every shadow and live model once an hour, and notifies the people who train and approve models when a version’s drift is significant (once a week for each version)	Without it, trainings stay Waiting to train the drift readings never change, and scheduled trainings are not queued (the schedules stay due). Scoring itself is done by the decision and stream roles. See Risk models for operators
Case deadlines	Every minute: finds open cases whose deadline has passed and records each missed deadline once: deadline missed on the case timeline, case.sla_breached in the audit log, a notification to the assignee, the case’s team and the people who approve closures, and the case.sla_breached webhook. When the case’s escalation policy says so and the case is below level 3, it then escalates the case one level. Handles up to 200 cases per workspace in a pass, and skips closed cases, closures awaiting approval and workspaces that are suspended or being moved. Logs cases: missed deadlines recorded with the count	Without it, cases still show as overdue in the queue, but nobody is told, nothing is recorded and no case escalates by itself
AI assistant	Every minute, for workspaces that switched on Review new alerts automatically: asks the AI assistant about open alerts that have no current advice, up to 10 per workspace in a pass, the most severe first. Stops for a workspace when its monthly allowance is used up, its key is refused or the assistant is switched off, and skips workspaces that are suspended or being moved. Logs assistant: alerts reviewed automatically with the count. Separately, it does the batches people ask for (Review the queue, Summarise the cases): it looks at the queue every 3 seconds, reads four alerts or cases at a time, takes back after 10 minutes an item a stopped worker left unfinished, and once an hour deletes items that finished more than 7 days ago	Without it, advice appears only when a person asks on an alert’s or case’s page, and a batch stays queued without progress. Nothing else depends on it
Approval requests	Every minute: closes approval requests past their 7-day expiry, writes change_request.expire and tells the maker. Skips workspaces that are suspended or being moved	Without it, overdue requests stay open in the list, though none can be approved after its expiry
API keys to rotate	Every minute: sets the gauge speculapro_api_keys_rotation_due	Without it, the gauge isn’t updated and its alert can’t fire. Keys are still marked when their creator is deactivated
Support access (hosted service)	Every minute: lapses requests nobody decided within 72 hours, ends accesses whose time is up with their sessions, writes the counted support.refused entries, and sets speculapro_support_access_active	Without it, a session is still refused the moment its time passes; only the record and the audit entry wait
Scheduled deletion (hosted service)	Every 10 minutes: looks for workspaces whose scheduled deletion date has passed. With SP_PROVIDER_PURGE_ENABLED it deletes those that are still suspended, under no legal hold, not being moved, approved by two people at least 30 days ago and whose reports were exported; otherwise it only counts them in speculapro_provider_purge_waiting	Without it, nothing is deleted on schedule. See Suspending and deleting

When it runs

The housekeeping (partitions and keys) runs once at start-up, then every SP_WORKER_INTERVAL (6 hours by default). Notifications run continuously; see below. Each run takes a few seconds on a normal installation. When it creates partitions it logs:

level=INFO msg="worker: created partitions" table=transactions count=1
level=INFO msg="worker: expired ingest keys" count=…

Seeing nothing on most runs is normal: there is usually nothing to do.

Running more than one

Each housekeeping run first takes a PostgreSQL advisory lock. If another worker holds it, the run is skipped silently. Notifications are shared out instead: each worker claims a batch of up to 50 messages for two minutes and only starts a message if it can finish within that time, handing the rest back, so no message is sent twice. If a worker stops, another picks up its messages when the claim runs out. Webhook deliveries are shared the same way, with a claim sized to the batch; see Running more than one worker. So you can run two worker processes for resilience without any risk of them clashing.

Notifications

When something happens that someone should hear about (a case or a screening match assigned, a closure to approve, an escalation, a critical alert), SpeculaPro writes a message to the notifications outbox in the same database transaction as the change itself. So a message exists if, and only if, the change was saved. One message is written per person and channel they have switched on. Which events notify whom is described in Notifications. Emails are laid out when they are sent, in the recipient’s language, with a plain-text part; see What an email looks like.

The worker then:

1. Checks the outbox every SP_NOTIFY_INTERVAL (5 seconds by default) and picks up to 50 messages that are due.
2. Sends each one through the configured provider: SMTP, the Spalce Notification Service, your GOWA server for WhatsApp, or the log. See Notifications settings.
3. Marks the result: sent; skipped when the channel isn’t configured (for example SMS with SP_NOTIFY_SMS_PROVIDER=none) or the person has no phone number; failed at once when retrying can’t help (WhatsApp says the number isn’t on WhatsApp); or, after any other failure, schedules a retry.
4. Retries with increasing waits: 30 seconds, 1 minute, 2 minutes, 4 minutes and so on, never more than an hour apart. After SP_NOTIFY_MAX_ATTEMPTS tries (6 by default, about 15 minutes in all) the message is marked failed. A WhatsApp message whose session is logged out, or whose GOWA device is missing, is different: it waits without using up tries, tried again every 30 seconds rising to every 5 minutes, and is marked failed only once it is older than SP_NOTIFY_WHATSAPP_MAX_WAIT (24 hours by default).
5. With WhatsApp through GOWA, checks every SP_NOTIFY_GOWA_STATUS_INTERVAL (1 minute) that the WhatsApp session is still logged in, for the metric speculapro_notify_whatsapp_session_up. When a worker starts with a different WhatsApp provider, GOWA address or device ID from last time, it writes notify.whatsapp_provider_changed to the audit log.
6. Every minute, looks for open cases due within the hour (or overdue by less than a day) and queues a warning for the owner, or for everyone with cases.approve when nobody owns the case. Each deadline is warned about once.

What you see in the logs:

level=INFO msg="notify: dispatcher started" component=notify email=smtp sms=none whatsapp=gowa
level=INFO msg="notify: (log channel)" channel=email [email protected] subject="Case SP-2026-000123 is assigned to you" link=…
level=WARN msg="notify: giving up" id=… channel=email attempts=6 err="smtp: dial smtp.yourbank.com.gh:587: …"
level=ERROR msg="notify: dispatch" err=…
level=WARN msg="notify: undeliverable" id=… channel=whatsapp err="gowa: /send/message: the number is not on WhatsApp (INVALID_JID)"
level=ERROR msg="notify: WhatsApp session is down: pair or reconnect the device in GOWA; WhatsApp notifications wait in the outbox meanwhile and fail if it stays down longer than SP_NOTIFY_WHATSAPP_MAX_WAIT" max_wait=24h0m0s
level=WARN msg="notify: giving up: the WhatsApp session was down for too long" id=… waited=24h0m30s err=…
level=INFO msg="notify: WhatsApp session is up"

The notify: (log channel) line appears only when a provider is set to log: the message was written to the log instead of being sent. Phone numbers in it are masked to their last three digits and one-time codes appear as [code] (unless SP_NOTIFY_LOG_SHOW_CODES=true, development only). Alert on notify: giving up; see Troubleshooting: notifications don’t arrive.

Messages carry no customer data

Email, SMS and WhatsApp leave your institution’s control, so messages contain only case and rule references, the name of the colleague who acted, and a link. Customer names, account numbers and amounts never appear. Mail headers are cleaned so that text from the app can’t add extra headers.

Backtests

A backtest replays up to 180 days of a workspace’s history through a rule version, in memory, and compares it with the live version. People start one on a rule’s page or in the rule builder; the API only queues it. A worker runs it in the background. Backtests never run on the API or decision path, and never read or write the live Redis/Valkey feature store.

With the optional analytics store, a run whose history (warm-up included) spans more than SP_CLICKHOUSE_BACKTEST_THRESHOLD (30 days) reads its transactions from ClickHouse instead of PostgreSQL. Before it does, the worker checks that the workspace’s copy is complete from the start of the run’s history and that the analytics role has caught up; see When a reader uses the store. If not, or if ClickHouse can’t be reached when the run starts, it reads PostgreSQL and the result says why. Every run records which of the two it read. The limits below apply either way.

A backtest of a scheduled rule works differently: it holds no feature memory and replays no transactions. It carries out every run of the rule’s schedule in the period, one after another, each as one read through the scenario’s own query, the same query a live scheduled run uses. Each simulated run has the statement timeout of a live one (5 minutes, or SP_BACKTEST_TIMEOUT if that is shorter), and reads from the backtest’s database connection, the read replica when SP_DB_REPLICA_URL is set. With the analytics store, the windows of I2 and I3 that are at least SP_CLICKHOUSE_SCENARIO_THRESHOLD long are read from ClickHouse, as for their live runs. Nothing is written but the result. See Backtesting a scheduled rule.

How runs are shared out

The queue is the backtests table. Every two seconds, each worker with free capacity takes the oldest queued run whose workspace is under its limit, while the whole installation is under its own. Claims are serialised with a PostgreSQL advisory lock, so the limits hold however many workers you run.

Limit	Setting	Default
Runs at once in one worker process	SP_BACKTEST_WORKERS	2. 0 turns backtests off in that process, leaving them to other workers
Runs at once across the installation	SP_BACKTEST_GLOBAL_LIMIT	4
Runs at once per workspace	SP_BACKTEST_TENANT_LIMIT	1
Runs queued or running per workspace	Fixed	5. A sixth is refused
Time per run	SP_BACKTEST_TIMEOUT	30m
Feature memory per run	SP_BACKTEST_MAX_MEMORY_MB	1024
Events per run, warm-up included	SP_BACKTEST_MAX_EVENTS	30000000: transactions, plus account events when the rule reads them. A larger period is refused before the run starts, after counting at most one more than the limit. For a scheduled rule it bounds the transaction rows its simulated runs read in all: a transaction inside several overlapping windows counts once per run, and the run fails at its start when the estimate is over the limit
Rows per history read	SP_BACKTEST_PAGE_SIZE	5000

The installation and workspace limits hold across replicas, even if workers are set up differently. Each worker records its own SP_BACKTEST_GLOBAL_LIMIT and SP_BACKTEST_TENANT_LIMIT when it claims (in the backtest_runners table), and a claim applies the strictest values among the workers seen in the last two minutes. Records of workers not seen for a day are removed.

A running backtest reports its progress every two seconds. That is also when it notices a cancellation. A run that hasn’t reported for two minutes (its process was killed or its machine lost) is marked failed with The run was interrupted (the worker stopped). Run it again. It is not retried automatically, so a run that brings its process down can’t bring the next one down too. When a worker shuts down cleanly, its runs are saved as failed with The worker stopped during the run.

What a run costs

• Database reads. A run reads the period plus its warm-up (up to another 180 days before it) in pages of SP_BACKTEST_PAGE_SIZE rows, in time order, fetching the next page while it replays the current one. These are long sequential reads. On a large installation, point them at a read replica with SP_DB_REPLICA_URL.
• Memory. Each run keeps the running totals of every customer, account and counterparty active in the period, for the windows the rule reads, plus customer profiles. If it grows past SP_BACKTEST_MAX_MEMORY_MB, the run stops with a clear message. Size the worker’s memory for SP_BACKTEST_WORKERS runs at that limit, plus its other work.
• CPU. One core per running backtest, roughly.

Indicative figures from a laptop

The developers measured, on a laptop, about 1 KB of feature memory per active entity (8.9 MB for 8,842 entities) and a replay rate of about 62,000 to 68,000 events per second end to end from PostgreSQL. These figures show the order of magnitude only. They are not guarantees, and your rules, data and hardware will give different numbers. Each finished run records its own rate and memory under How this was computed.

What you see in the logs

level=INFO msg="backtest runner started" component=backtest workers=2 global_limit=4 tenant_limit=1
level=INFO msg="backtest finished" component=backtest backtest=… tenant=… status=succeeded events=…
level=WARN msg="backtest: run interrupted, marked failed" component=backtest backtest=… tenant=…
level=ERROR msg="backtest: run failed" component=backtest backtest=… tenant=… err=…

backtest: run failed is logged when a run fails for an unexpected reason; the person who started it sees The run failed on our side; it has been logged. With SP_BACKTEST_WORKERS=0 the worker logs backtest runner disabled in this process (SP_BACKTEST_WORKERS=0) instead.

Keep at least one worker with backtests switched on

If every worker runs with SP_BACKTEST_WORKERS=0, backtests are accepted but stay Waiting for a worker until one can run them.

What key clean-up means for you

When you send a transaction, SpeculaPro records its external_id so that a resend is recognised as a duplicate. The worker deletes these records after the retention period.

After 90 days, a resent transaction counts as new

Once a transaction’s key has been deleted, sending the same external_id again is treated as a new transaction and stored a second time. Duplicate protection is designed for retries within hours or days, not for re-sending old history.

If you need to reload old data (for example after a migration on your side), check first which transactions SpeculaPro already holds, and send only those that are missing. If your systems may legitimately retry after a longer gap, raise SP_WORKER_INGEST_KEY_RETENTION, for example 4320h for 180 days, at the cost of a larger key table.

Only keys whose transactions were confirmed by Kafka are deleted. A key whose publication was never confirmed is kept, so a later retry still reuses the same transaction ID.

Settings

Setting	Default	Meaning
SP_WORKER_INTERVAL	6h	Time between runs
SP_WORKER_INGEST_KEY_RETENTION	2160h (90 days)	Age after which confirmed idempotency keys are deleted
SP_NOTIFY_*		How messages are sent; see Notifications settings
SP_BACKTEST_*, SP_DB_REPLICA_URL		Backtest limits and an optional read replica; see Backtests and Backtests settings
SP_DB_SYSTEM_URL	SP_DB_URL	The worker handles every workspace, so in production it must connect as speculapro_system. See Database roles

Keep at least one worker running

If no process runs the worker role, nobody receives notifications, partitions stop being created after the three months the migrations made, and idempotency keys pile up. The production checklist includes this. If you really can’t run one, see Not running the worker role?

Related

• Running roles
• PostgreSQL: partitions
• Idempotency and retries