Kafka / Redpanda

SpeculaPro moves every event (a transaction arriving, a decision made, a rule firing, an alert created) through an event log. Kafka is the best-known event log; Redpanda is a compatible alternative that is simpler to operate. SpeculaPro works with either, because it speaks the standard Kafka protocol.

Why an event log?

• Nothing is lost. An event is written to disk on several machines before SpeculaPro acknowledges it. If a worker crashes, the event waits until a worker picks it up again.
• Order per customer. Events are keyed by workspace and customer or account. Every event for the same account goes to the same partition and is processed in order, so running totals are always right without locking.
• Spikes are absorbed. On salary day or at month-end, events queue in the log instead of overloading the database.
• Replay. Events can be re-read from any point, for example to rebuild the feature store.

Kafka or Redpanda?

	Redpanda	Apache Kafka
Operation	One binary, no JVM, no ZooKeeper	JVM; KRaft mode in recent versions
Small installations	Easiest: a single process	More moving parts
Licence	Source-available (Business Source Licence) for the core; check it suits your use	Apache 2.0
Ecosystem and managed offerings	Smaller	Very large (Confluent, MSK, Aiven, Strimzi on Kubernetes)

The local development stack uses Redpanda. In production, choose whichever your team can run well. If you already operate Kafka, use it.

Official documentation: Redpanda · Apache Kafka.

Topics

SpeculaPro uses eight topics, each with a dead-letter twin ending in .dlq. All names carry the SP_KAFKA_TOPIC_PREFIX (default sp.).

Topic	Carries	Status
sp.txn.ingested	Transactions, keyed by workspace and account. A transaction sent for a real-time decision arrives here too, carrying its decision	In use: written by ingest and decision, read by persist and by stream
sp.account.events	Account events (sign-ins, failed one-time codes, SIM swaps, profile changes), keyed by workspace and customer	In use: written by ingest, read by persist (stored for the customer’s activity) and by stream (fed into the running totals rules read)
sp.txn.decided	Real-time decisions	Created now, not used: decisions travel on sp.txn.ingested with their transaction
sp.entity.updated	Customer, account and counterparty changes	Created, not used: the ingest API writes customers and accounts straight to Postgres
sp.rule.hit	Rule hits	Created, not used: the stream evaluator writes hits and alerts straight to Postgres
sp.alert.created	New alerts	Created, not used: alerts are written straight to Postgres, with their notifications and webhooks queued in the same transaction
sp.case.events	Case activity	Created, not used: case events are written straight to Postgres
sp.notify.requests	Notifications to send	Created, not used: notifications are queued in Postgres and sent by the worker

A dead-letter queue (DLQ) is where a message goes if it cannot be processed, for example because it is malformed. It is parked for inspection instead of blocking everything behind it or being thrown away. Two are in use: sp.txn.ingested.dlq and sp.account.events.dlq, both written by the persister. See Persister: dead letters.

Consumer groups

Kafka tracks how far each reader has got through a topic by consumer group. Two groups read sp.txn.ingested, each getting every transaction, and two more read sp.account.events:

Group	Setting	Topic	Reader	Its lag tells you
speculapro-persist	SP_PERSIST_GROUP	sp.txn.ingested	Persister	Whether stored transactions and decisions are keeping up with what was accepted. Decision webhooks are queued as decisions are stored, so lag here delays them too
speculapro-stream	SP_STREAM_GROUP	sp.txn.ingested	Stream evaluator	Whether rules and alerts are keeping up. Lag here means alerts arrive late
speculapro-analytics	SP_CLICKHOUSE_GROUP	sp.txn.ingested	Analytics store, only where it is set up	Whether the optional ClickHouse copy is keeping up. Lag here delays only the Analytics page and long backtests
speculapro-persist-events	SP_PERSIST_GROUP followed by -events	sp.account.events	Persister	Whether stored account events (the customer’s activity screen) are keeping up
speculapro-stream-events	SP_STREAM_GROUP followed by -events	sp.account.events	Stream evaluator	Whether account events are reaching the running totals through Kafka. The ingest API also applies each event straight away, so rules usually see it before this group does

Every process of one role uses the same group, so Kafka divides the partitions between them. The two names must be different: if persist and stream shared a group, each transaction would go to only one of them.

Renaming a group replays the whole topic

A new group name starts from the beginning of the topic.

• Persister: nothing is duplicated, because every write is idempotent, but everything still retained is re-read, which can take a while.
• Stream evaluator: no hit or alert is created twice. But every transaction older than the replay window (SP_STREAM_REPLAY_TTL, 30 minutes by default) is added to the running totals a second time, because SpeculaPro only remembers that long that it has already counted a transaction. Rules that look back over days or months would then see inflated totals. See Stream evaluator: replays.

Change SP_PERSIST_GROUP or SP_STREAM_GROUP only deliberately.

How to create the topics

speculapro topics

Run it before starting the ingest and persist roles. The command connects using the SP_KAFKA_* settings, creates any of the 16 topics that are missing, and prints created <name> for each. Topics that already exist are left completely untouched, so the command is safe to run on every deployment.

Turn off automatic topic creation

Many brokers create a topic automatically the first time something writes to it, using the broker’s defaults (often 1 partition, 1 replica). On production clusters, switch automatic creation off (auto.create.topics.enable=false on Kafka, auto_create_topics_enabled: false on Redpanda) and create topics only with speculapro topics, so every topic gets the right partitions and replicas.

After upgrading

New versions can add topics. Version with migration 00013_realtime adds sp.account.events and its dead-letter twin. Run speculapro topics after every upgrade, before starting the new version; with automatic topic creation switched off, sending account events fails until the topic exists.

When Kafka is down

What happens depends on the role:

Role	While Kafka is unreachable
ingest	Items it can’t hand to Kafka within 10 seconds are answered retry; your system sends them again later. Nothing is accepted that isn’t safely in Kafka
decision	Decisions are still answered, on time. Each decided transaction waits in a large in-memory buffer for up to 30 seconds; if Kafka still can’t take it, it is parked in the spool, a list in Redis/Valkey (spool:{decisions}). Every second, the decision role tries to replay the spool into Kafka, oldest first
persist, stream	Wait, and carry on from where they stopped when Kafka is back

While decided transactions wait in the spool they aren’t yet stored, monitored or sent as webhooks, and they don’t appear on the Decisions page. They arrive late, but they arrive: storing is idempotent, so a replayed transaction is never stored twice.

Stopping a decision node doesn’t lose its buffer: on a clean stop it gives Kafka up to 10 seconds, then parks whatever is left in the spool before closing Redis. See What happens on shutdown.

A decided transaction is lost only if Kafka and Redis both fail at the same time, or if a spool entry turns out to be unreadable. Watch these metrics:

• speculapro_decision_events_spooled_total: transactions parked;
• speculapro_decision_events_unspooled_total: parked transactions delivered later;
• speculapro_decision_spool_events: how many are waiting right now (zero normally);
• speculapro_decision_events_lost_total: lost. Alert on any increase. The log line decision: event LOST names each one’s workspace and external_id, so you can send it again through the ingest API.

Partitions

A topic is split into partitions. Each partition is processed by one worker at a time, so the number of partitions is the maximum number of workers that can share the load of a topic.

SpeculaPro creates topics with 64 partitions by default (SP_KAFKA_PARTITIONS). That is more than a small installation needs today, deliberately:

Choose the partition count before go-live

Because events are keyed by customer or account, adding partitions later changes which partition each customer’s events go to. Events for the same customer could then be processed out of order during the change. Pick a number that covers your growth for years. 64 is the recommended default; for a laptop, 3–6 keeps things light.

Changing SP_KAFKA_PARTITIONS later has no effect on topics that already exist.

How many partitions the broker accepts

speculapro topics creates 16 topics: the eight in the table above and a dead-letter twin (.dlq) for each, all with SP_KAFKA_PARTITIONS partitions. So the broker must accept 16 × SP_KAFKA_PARTITIONS partitions for one SpeculaPro environment, times the replication factor in replicas across the cluster:

SP_KAFKA_PARTITIONS	Partitions created
6	96
16 (the production Compose default)	256
64 (the default)	1,024

Brokers limit how many partitions they hold. Redpanda allows about 1,000 per CPU core by default, so 64 partitions per topic doesn’t fit a single-core broker such as the local development container, or the production Compose stack on its default of one core, and it uses a large share of a small shared cluster. When the limit is reached, topic creation fails part-way: some topics exist and the rest are refused.

• On a laptop or a shared test broker, use a small number, 3 to 6.
• Sharing one cluster between environments (different SP_KAFKA_TOPIC_PREFIX values), add up every environment’s 16 × its partition count.
• In production, size the cluster for the count you chose before go-live: enough cores across the brokers for 16 × partitions × replication factor, with room for the broker’s own topics. Check your broker’s documentation for its exact limit.

Replication

The replication factor is how many brokers keep a copy of each partition.

Setting	Meaning	Use
-1 (default)	The broker’s own default	Development; on a single node this is 1
3	Three copies; survives losing a broker	Production

Set SP_KAFKA_REPLICATION_FACTOR=3 before running speculapro topics on a production cluster. Also set the brokers’ min.insync.replicas to 2, so a write is only acknowledged once two copies exist.

Listeners and addresses

Clients first connect to the address in SP_KAFKA_BROKERS, then the broker replies with the address it advertises, and clients use that from then on. This is the most common cause of connection problems.

The local stack advertises two addresses:

Listener	Address	For
internal	redpanda:9092	Other containers on the Compose network
external	localhost:19092	Programs on your computer, including make run

So on your computer use localhost:19092; a SpeculaPro container in the same Compose project would use redpanda:9092. In production, make sure the advertised address is resolvable and reachable from every SpeculaPro server.

Security

Need	Settings
Encryption in transit	SP_KAFKA_TLS_ENABLED=true, plus SP_KAFKA_TLS_CA_FILE if the brokers use a private certificate authority
Username and password	SP_KAFKA_SASL_MECHANISM = SCRAM-SHA-512 (recommended), SCRAM-SHA-256 or PLAIN, with SP_KAFKA_USERNAME and SP_KAFKA_PASSWORD
Client certificates (mutual TLS)	SP_KAFKA_TLS_CERT_FILE and SP_KAFKA_TLS_KEY_FILE

PLAIN sends the password as-is

Only use SASL_MECHANISM=PLAIN together with TLS. SCRAM never sends the password itself.

TLS 1.2 is the minimum version SpeculaPro accepts.

Inspecting topics

In development, Redpanda Console runs at http://localhost:8081. You can browse topics, read messages and see consumer lag there.

From the command line, with Redpanda’s rpk:

rpk group describe speculapro-persist          # lag per partition for the persister
rpk group describe speculapro-stream           # lag per partition for the stream evaluator
rpk topic consume sp.txn.ingested.dlq -n 10    # read the first dead letters

Messages are sensitive

Events on sp.txn.ingested carry identifiers such as phone and account numbers only in encrypted form, with masked versions for display. But counterparty names travel as sent, and amounts, dates, references and decisions are all there. Restrict who can read the topics, as you would the database.

Related

• Configuration reference: Kafka
• Components and Kafka topics
• Troubleshooting: Kafka