Stream evaluator

In short

The stream evaluator (role stream) is where monitoring actually happens. For every transaction that arrives, it:

1. adds the transaction to the running totals of its account, customer and counterparty,
2. reads the totals your rules ask for, as of that transaction,
3. runs every enabled rule,
4. records each hit, with the exact values it was based on, and
5. turns new hits into alerts, one open alert per customer, account or counterparty, unless a suppression applies.

It reads the same Kafka topic as the persister (txn.ingested), in its own consumer group, so the two work side by side: one files transactions away, the other checks them.

Totals build up even before any rule is enabled

A workspace with no enabled rules gets no hits or alerts, but every transaction still updates the running totals. So when you enable your first rule, it already has the history collected since you started sending data. See Managing rules.

What happens to each batch

The evaluator takes up to SP_STREAM_BATCH_MAX transactions at a time (default 2,000), waiting at most a tenth of a second for more.

Step	What it does	Where
Load the workspace’s setup	Enabled rules (active and shadow versions), lists, approved, live suppressions, time zone and base currency	Cached, refreshed every 15 seconds
Look up the parties	The customer, account and counterparty of each transaction, with their current risk rating, KYC level, active risk designations and custom attributes. Creates placeholder records for any not seen before	PostgreSQL
Update and read totals	Adds each transaction to the feature store and reads the features the rules declare, all in one ordered round trip, so each transaction sees exactly those before it	Redis/Valkey
Evaluate	Runs every enabled rule on every transaction	In memory
Record	Writes every hit, and merges new live hits into alerts	PostgreSQL, one transaction for the whole batch
Acknowledge	Tells Kafka the batch is done	Kafka, after the database commit

Changes take about 15 seconds to apply

The evaluator re-reads each workspace’s setup every 15 seconds. Enabling or disabling a rule, deploying a new version, adding a list item or approving a suppression therefore takes effect within about 15 seconds, on every stream process. Transactions evaluated in between use the previous setup. This interval is fixed.

Who an alert is about

Each rule declares its entity scope. The hit, and so the alert, is about:

Rule’s scope	Subject of the hit and alert
account	The transaction’s account
counterparty	The transaction’s counterparty
Anything else (customer, transaction, …)	The transaction’s customer

If the transaction has no account (or no counterparty), an account-scoped (or counterparty-scoped) rule falls back to the customer.

Alerts

New, live hits are merged into one open alert per subject. A second hit on the same customer does not create a second alert; it is added to the first. How the merge works is described in Alerts and cases.

If the subject has an active case, the evaluator works on the case instead, in the same database transaction:

• New hits extend the case’s most recent alert (hit count, exposure, rules, and severity if higher). Only if the case has no alert attached is a new alert created, and it is linked to the case straight away.
• If the hits are more severe than the case’s priority, the case’s priority is raised and its SLA deadline is brought forward if that is sooner.
• A new link or a priority raise writes an alert_linked event on the case timeline and a case.link_alert entry in the audit log, both by the system.

See Alerts that join a case automatically.

Hits are always recorded, but no alert is raised for:

• hits from a shadow version (they are there to compare, see shadow mode),
• hits covered by an approved suppression that has started and not expired or been revoked (the hit records which suppression applied); a suppression still waiting for approval has no effect, and
• hits from a rule whose action is tag, score_only or allow (see Rule engine: what a rule produces).

If the subject already has a case that isn’t closed, a new alert goes straight into that case instead of the queue, and the case’s timeline records that it was linked automatically. See Alerts and cases.

Replays and retries

Kafka may deliver a transaction more than once: after a crash, a restart or a rebalance between evaluators. That is harmless:

• Hits are never duplicated. A hit’s ID is derived from the transaction and the rule version, so a second delivery finds it already recorded.
• Totals are not counted twice. The feature store remembers which transactions it has already counted, for the replay window (SP_STREAM_REPLAY_TTL, 30 minutes by default).
• The same answer comes out. The feature values each transaction was evaluated with are kept for the same window and reused, so a re-delivered transaction produces exactly the same hits even if later transactions have moved the totals on.

Kafka only re-delivers transactions whose batch was never confirmed, so the window has to cover the time between a batch being read and the evaluator coming back after a failure, not your history.

Sizing the replay window

Set SP_STREAM_REPLAY_TTL longer than the longest stream outage you expect to recover from without intervention, for example a Redis or PostgreSQL maintenance window. Every transaction keeps two small Redis entries for the whole window, so Redis memory grows with transactions per second × window: at 5,000 transactions per second, a 30-minute window holds about 9 million transactions’ worth of entries. The minimum is one minute: a smaller value (for example 30s) stops every SpeculaPro process from starting, with config: SP_STREAM_REPLAY_TTL must be at least 1m, got 30s.

Don’t replay history

If the evaluator is down for longer than the replay window, the batch it was working on is counted again when it comes back. Resetting or renaming the stream consumer group (SP_STREAM_GROUP) is worse: the evaluator re-reads everything Kafka still retains. For any transaction older than the replay window:

• they are added to the running totals a second time, inflating long-window features for later transactions;
• their saved feature values have expired, so they are evaluated against today’s totals;
• if a rule has a new version since, that version has never seen them, so it can raise new hits and alerts on old activity.

Change the group only deliberately, and only after thinking through these effects.

When a batch keeps failing

If a batch can’t be completed (PostgreSQL or Redis down, a permission missing), the evaluator logs

level=ERROR msg="stream: batch failed, retrying" events=… err=… backoff=…

and retries the same batch, waiting longer each time, up to 30 seconds between attempts. Nothing is skipped and nothing is lost, but the partitions that evaluator owns stop moving, and alerts arrive late. Transactions keep arriving and waiting on Kafka.

Kafka retention is your safety margin here too

Transactions waiting longer than the topic’s retention period are deleted by Kafka and never evaluated. Alert on repeated batch failed messages and on growing lag. See Backups: Kafka.

A rule that errors

If one rule fails on one transaction (for example, a custom rule reads a custom attribute that the transaction doesn’t carry), the evaluator logs a warning, skips that rule for that transaction, and carries on:

level=WARN msg="stream: rule error" tenant=… rule=custom.my_rule err=…

Other rules and other transactions are not affected. A rule that errors on every transaction fills the log, so fix it, or disable it. Rules from SpeculaPro’s own packs check that an attribute exists before reading it, so they never error this way. A list that a rule names but your workspace hasn’t created is not an error: it reads as empty. See Lists in rules.

Unreadable records

Records the evaluator can’t read are skipped. The persister reads the same records and moves them to the dead-letter topic, so they are dealt with there.

Starting up

Like the persister and the worker, the stream evaluator reads and writes every workspace at once, so it must connect to PostgreSQL as speculapro_system (or another user with BYPASSRLS), set in SP_DB_SYSTEM_URL. At start-up it checks, and if the user can’t bypass row-level security it refuses to start:

error: persist/stream/worker roles: database role cannot bypass row-level security (set SP_DB_SYSTEM_URL to the speculapro_system role)

See Database roles.

When it starts, it logs one line per evaluator, then one line per workspace once that workspace’s rules are loaded:

level=INFO msg="stream evaluator started" stream_worker=0 topic=sp.txn.ingested
level=INFO msg="stream: rules loaded" tenant=… rules=12

A brand-new consumer group starts from the beginning of the topic, so the first start of a stream evaluator on an installation that already has data evaluates everything Kafka still retains (with your current rules).

Settings

Setting	Default	Meaning
SP_STREAM_GROUP	speculapro-stream	Kafka consumer group. The same on every stream process of one installation; different from SP_PERSIST_GROUP
SP_STREAM_BATCH_MAX	2000	Most transactions evaluated in one batch
SP_STREAM_WORKERS	4	Parallel evaluators in this process
SP_STREAM_REPLAY_TTL	30m	How long a re-delivered transaction is still recognised (see Replays and retries). Longer than your longest expected outage; Redis memory grows with throughput × this value
SP_DB_SYSTEM_URL	SP_DB_URL	Must point at speculapro_system in production

The evaluator also needs the usual SP_REDIS_* settings: it is the main user of Redis/Valkey.

Sizing

Kafka gives each partition of txn.ingested to exactly one evaluator in the group. The total number of evaluators across all stream processes (processes × SP_STREAM_WORKERS) should therefore not exceed the partition count (64 by default); any beyond that sit idle.

Stream processes	SP_STREAM_WORKERS	Evaluators	With 64 partitions
1	4	4	Small installations
2	8	16	Fine
4	16	64	The most that do useful work

What limits throughput:

• Rules and features. Every enabled rule runs on every transaction, and every declared feature is a Redis read. More rules with more features cost more per transaction.
• Redis/Valkey latency. Each batch makes one round trip per workspace; place stream processes close to Redis.
• PostgreSQL. Each batch is one database transaction that looks up the parties and writes hits and alerts.

The stream evaluator has not been benchmarked yet. Measure with your own rules and volumes; see Load testing.

Monitoring

Stream-only processes have no HTTP port, so there is no health endpoint to poll. Watch instead:

Signal	Healthy	Investigate when
Lag of the speculapro-stream consumer group	Near zero, short spikes at busy times	It keeps growing: alerts are arriving late. Add evaluators (up to the partition count) or find the slow dependency
stream: batch failed, retrying	Absent	It repeats: the evaluator is stuck on the error it logs
stream: rule error	Absent	It appears: a rule is broken; fix or disable it
Redis/Valkey memory	Stable over a week	It keeps growing: see Feature store: memory

rpk group describe speculapro-stream      # lag per partition

See Monitoring.

Troubleshooting

Symptom	Likely cause	What to do
Transactions are stored but no hits or alerts appear	No enabled rules in the workspace, the rules’ conditions are not met, or the rules’ action is tag, score_only or allow (hits only)	Check rules are installed and enabled. Rules that read a list don’t fire until the list has items
Hits appear but no alerts	The hits are from shadow versions, or a suppression applies	Check the hit’s shadow flag and suppression. See Shadow mode and Suppressions
A rule change seems ignored	Less than 15 seconds have passed, or the new version was approved but not deployed	Wait 15 seconds; check the version is in the active slot
Long-window rules fire less than expected	Totals only started building when the first rule was enabled, or the feature store was emptied	See Feature store
Process exits at start-up with cannot bypass row-level security	SP_DB_SYSTEM_URL missing or pointing at the wrong user	See Starting up

Related

• Feature store
• Alerts and cases
• Managing rules
• Running roles
• Persister
• Kafka / Redpanda: consumer groups