Components and Kafka topics

In short

Inside SpeculaPro, work is split between specialist components, like stations on a factory floor. They don’t call each other directly. They pass work along a conveyor belt, the message stream (Kafka or Redpanda). Each kind of item travels on its own labelled belt, called a topic.

This design means a slow station never stops the others. If the station that writes to the database falls behind, the conveyor simply holds the items until it catches up, and nothing is lost.

The whole picture

                ┌──────────────┐  sync  ┌─────────────────┐
  PSP / Bank ──▶│ Decision API │───────▶│ Rule engine     │◀── Feature store (Redis/Valkey)
                └──────┬───────┘        │ (CEL, compiled) │
                       │ decided        └─────────────────┘
                       │ transaction (via a Redis spool if Kafka is down)
  Ingest API / files ──┤
                       ▼
               ┌──────────────────── Kafka / Redpanda ───────────────────┐
               │ txn.ingested · account.events                           │
               └───┬──────────────────────┬──────────────────────────────┘
                   ▼                      ▼
            Stream evaluator          Persister
            (counters, post-event     (COPY → Postgres; queues
             rules, alerts)            decision webhooks)
                   │                      │
                   ▼                      ▼
                   Postgres ──▶ Worker: notifications (email, SMS)
                            ──▶ Worker: signed webhooks to your systems
                   │
                   ▼
            Case management API ◀──▶ Web app

Read it top to bottom: transactions come in at the top, are checked and counted in the middle, and end up as alerts and cases for your team at the bottom.

The components

Component	Plain-language job	Technical detail	Role
Decision API	Answers “should this payment go through?” before it is approved	Validates, enriches from cache, reads features, evaluates realtime rules, returns a decision with reasons within the workspace’s time budget, then hands the decided transaction to txn.ingested without making the caller wait. If Kafka can’t take it, it waits in a Redis spool. See The decision role.	decision (available)
Ingest	The front door for data that doesn’t need an instant answer	The ingest API for transactions, customers, accounts, account events and FX rates: validation, duplicate protection, currency conversion, encryption of personal data. Transactions go to txn.ingested and account events to account.events; customers, accounts and rates are written straight to Postgres. File uploads come through the web app; Kafka topics and file drops through the connector.	ingest (available)
Connector	Collects data institutions don’t push	Reads institution Kafka topics as a consumer group and fetches file drops from SFTP servers and S3 buckets (CSV, JSON Lines, ISO 20022), and follows database changes (change data capture: Debezium topics, or Postgres logical replication through a replication slot), then loads them in process through the same ingest service. Rejected Kafka records go to a dead-letter topic; each file gets a results file; change data capture keeps a checkpoint and a ledger of applied changes so a restart neither loses nor repeats anything. Outbound connections go through an address check. See Connector.	connector (available)
Feature updater	Keeps running tallies per customer, account and counterparty	Reads txn.ingested and updates the counters in the feature store, as of each transaction, including the averages and hour-of-day counts that “unusual for this customer” rules read.	stream (available)
AML evaluator	Looks for patterns that take days or months to show	Evaluates the enabled post-event rules on every transaction, with counters looking back up to 180 days. Scheduled rules (patterns across many transactions, such as mule networks and circular flows) run on a timetable in the worker. See Stream evaluator.	stream (available); scheduled rules: worker (available)
Persister	Files everything away permanently	Reads txn.ingested and writes transactions (and any counterparties, customers or accounts they mention for the first time) to Postgres in large batches with COPY. It also stores each decided transaction’s decision, and queues its webhooks in the same database transaction. A second reader stores account events from account.events. Rule hits and alerts are written by the stream evaluator itself. See Persister.	persist (available)
Analytics writer (optional)	Keeps a second copy for questions that span many months	Only where an installation has the optional analytics store (ClickHouse). Reads txn.ingested in its own consumer group and writes transactions and decisions to ClickHouse in large batches; copies rule hits, alerts and cases from Postgres every 30 seconds. Postgres stays the system of record: the copy can be rebuilt from Postgres or Kafka at any time. Long backtests, two long scheduled rules and the Analytics page read it, and fall back to Postgres when it is absent or down.	analytics (available, optional)
Risk models (optional)	Give each transaction an anomaly score rules can read	A workspace can train an anomaly model on its own transactions. The worker trains it by replaying history through the backtest engine and measures its drift; the decision API and the AML evaluator score each transaction with the workspace’s live model in the same step as the rules, and the evaluator also scores with a shadow model for comparison. Models live in Postgres. A score never decides anything: only a rule that reads it can	worker (training, drift), decision, stream (scoring)
Alert manager	Turns rule hits into a manageable alert queue	Part of the stream evaluator: records every hit, applies suppressions, and merges hits into one open alert per subject, and adds new activity on a subject under investigation to its active case. See Alerts and cases.	stream (available)
Case API	Where investigations happen	Cases, assignment, notes, attachments, escalations, SLAs, dispositions, STR/SAR drafting and export.	api
Notifier	Tells people when something needs them	Email (SMTP or notify.spalce.net) and SMS (notify.spalce.net), from templates in English and French. WhatsApp through your own GOWA server; see WhatsApp through GOWA. See Notifications.	worker (available)
Webhook dispatcher	Tells your other systems when something happens	Delivers signed events (blocked and held payments, new alerts, case changes) from an outbox written in the same transaction as the change, with retries for about 27 hours. See Webhooks.	worker (available)
Admin and configuration API	Settings for your institution	Users and roles, rule packs, rules, lists, custom fields, thresholds, FX rates, approvals.	api
Web app	What your team sees	Dashboards, rule builder, alert queue, case workspace, reports, administration.	api
Directory (multi-region hosting only)	Tells the sign-in page which region a workspace is in	A separate small service with its own database, holding workspace slugs and regions and no personal data. Each region is a complete, independent copy of everything above. See Multi-region	None: started with speculapro directory run

What runs today, and what is still being built

All seven roles run today: payments are decided in real time, data is sent in, encrypted, stored, counted, checked against your enabled rules, and turned into alerts that your team works as cases, and people and systems are told by notification and webhook. The optional eighth role, analytics, runs only where the analytics store is set up. See roles and their status.

The worker role also does housekeeping: it creates future database partitions, clears out old idempotency keys and deletes old webhook history. For workspaces that switched it on, it runs the retention purge, which deletes or pseudonymises data past its period and drops old monthly partitions when every row in them may go, and it carries out scheduled erasures from data subject requests. See Worker.

Topics: the labelled conveyor belts

Every event carries a key made of your workspace ID and the customer or account it concerns (tenant_id:entity_id). For txn.ingested that is your workspace ID and your own account reference (tenant_id:account_external_id), so all of one account’s transactions stay in order. Kafka guarantees that events with the same key always travel on the same partition, in the order they happened.

That is what lets SpeculaPro keep an accurate running total per account without complicated locking: all of one account’s transactions are handled one after another, by one worker.

Running totals per customer add up all the customer’s accounts. A customer’s accounts can sit on different partitions and be processed in parallel, so two transactions on two different accounts of the same customer, arriving at almost the same moment, may each be evaluated without the other in the customer’s total yet. Totals per account are always exact.

Topic	What travels on it	Who puts it there	Who reads it
txn.ingested	Every transaction received, including those sent for a real-time decision (with their decision)	Ingest and the decision API	Persister and stream evaluator, each in its own consumer group
account.events	Sign-ins, failed one-time codes, SIM swaps, profile changes and other account events, keyed by workspace and customer	Ingest	Persister (stores them) and stream evaluator (counts them), each in its own consumer group
txn.decided	Reserved for real-time decisions	Not used: decisions travel on txn.ingested with their transaction
entity.updated	Changes to customers, accounts and so on	Not used: the ingest API writes customers and accounts straight to Postgres
rule.hit	Every time a rule fires	Not used: the stream evaluator writes hits straight to Postgres
alert.created, case.events	New alerts and case activity	Not used: alerts and case events are written straight to Postgres, with their notifications and webhooks queued in the same transaction
notify.requests	Messages to send	Not used: notifications are queued in Postgres
*.dlq	Events a component could not process	Each component (today: the persister, into txn.ingested.dlq and account.events.dlq)	People, for investigation and replay
Institution topics	Records your systems publish for SpeculaPro to read, one item each, and their dead-letter topics (<topic>.dlq)	Your systems	The connector. See Kafka topics
Debezium topics	Change events Debezium publishes for your core-banking tables, and their dead-letter topic	Your Debezium (Kafka Connect)	The connector, through the table mapping. See Change data capture

Topic names carry a prefix

In a running installation each topic name starts with a configurable prefix, sp. by default, so txn.ingested appears in Kafka as sp.txn.ingested. The speculapro topics command creates any missing topics, plus their dead-letter twins. See Self-hosting. Institution topics are named by you and don’t carry the prefix; SpeculaPro never lets a workspace read its own sp. topics.

Dead-letter topics

If a component cannot process an event (say, a malformed record), it does not block the conveyor or silently drop the item. It moves the event to its dead-letter topic (*.dlq) and carries on. Someone can then look at what went wrong, fix it, and replay the event.

Why it’s built this way

• Isolation of failures. A slow database or a burst of alerts slows only the component concerned. The conveyor absorbs the backlog.
• Replay. Kafka keeps events for a period of time. If a bug is fixed, or a derived store has to be refilled, events can be processed again from an earlier point.
• Independent scaling. Each component scales on its own. More stream workers for a busy month-end; more decision nodes for a flash sale.
• Works with both Kafka and Redpanda. The client library (franz-go) supports both, so you can use what your organisation already runs.

What this means for you

• Operations: watch consumer lag (how far each component is behind the conveyor) and the dead-letter topics. Those are the early warnings.
• Capacity planning: topics are created with enough partitions for the target load from day one (64 by default), because adding partitions later would break per-customer ordering. See Scaling and performance.

Related

• A real-time decision, step by step
• Feature store
• Alerts, cases and escalation