Scaling and performance

In short

SpeculaPro is designed to check thousands of payments per second, answer real-time checks in a few milliseconds, and keep working when individual servers fail. It grows by adding servers, not by buying ever-bigger ones.

These are targets, not measurements

SpeculaPro is pre-release. The figures below are the engineering targets the system is being built and tested against. They will be confirmed, and published, against a pilot institution’s real volumes.

Performance targets

What	v1 target	Design ceiling (by adding servers)
Real-time decisions per second	5,000 sustained; 15,000 peak (3× bursts on month-end and salary days)	50,000+
Time to answer a real-time decision (server-side)	99% within 50 ms; half within 10 ms	Same
From receiving a transaction to an analyst seeing the alert	95% within 5 seconds	Same
Batch file loading	1 million rows in 10 minutes or less	Grows with the number of workers
History available to rules	180 days immediately; years in archive
Availability of the decision API (SaaS)	99.9%	99.95%

Reading the numbers

• TPS means transactions per second.
• p99 ≤ 50 ms means 99 of every 100 answers arrive within 50 milliseconds. The remaining 1% may take longer.
• 99.9% availability allows about 43 minutes of downtime a month; 99.95% allows about 22.

How SpeculaPro grows

 small institution                         large institution
┌─────────────────┐                ┌──────────┐ ┌──────────┐ ┌──────────┐
│ speculapro      │                │ decision │ │ decision │ │ decision │  ← more for peak payments
│   --role=all    │                └──────────┘ └──────────┘ └──────────┘
└─────────────────┘                ┌──────────┐ ┌──────────┐
  1 Postgres                       │ stream   │ │ stream   │  ...up to the partition count
  1 Redpanda                       └──────────┘ └──────────┘
  1 Valkey                         ┌──────────┐ ┌──────────┐ ┌──────────┐
                                   │ persist  │ │ api      │ │ worker   │
                                   └──────────┘ └──────────┘ └──────────┘
                                     Postgres HA · Kafka cluster · Redis Cluster

• Stateless components scale sideways. Run more copies of a role behind a load balancer.
• Stream workers scale up to the partition count. Kafka gives each partition to one worker at a time, so 64 partitions allow up to 64 workers per consumer group.
• Postgres writes are batched by the persister using COPY, aiming for at least 50,000 rows per second per persister.

First measurements

The ingest API and persister have been measured on a single laptop (everything on one MacBook, in-memory Kafka, untuned PostgreSQL 18). Those are indicative development measurements, not capacity claims, and the persister is still well short of its target per node. The real-time decision API has been load-tested on a laptop too. Figures will be published only once measured on production-like hardware. See Load testing to measure your own setup.

What limits real-time decisions

The decision service itself keeps no state: add copies behind a load balancer. What limits how many decisions a second can be answered within budget is Redis/Valkey.

• Each decision costs a few hundred Redis commands. With the demo workspace’s rule packs, about 400, counting the post-event monitoring of the same transaction. Most run inside scripts, so a decision needs only about 15 round trips. The count depends on your rules: a rule window reads one time bucket for each hour or day it covers.
• The keys are grouped for sharding. Every running-total key carries a hash tag for its workspace and one customer, account, counterparty or device. On Redis or Valkey Cluster, one entity’s totals live together on one shard, and different entities spread evenly over all of them. Adding shards adds capacity close to linearly.
• Decisions get their own connections. The decision role keeps a separate Redis connection pool, so other work in the same process can’t make a live decision wait.
• Load is shed, not queued. Past SP_DECISION_MAX_INFLIGHT decisions in progress, a process answers new requests at once with the workspace’s failure policy, so a slow dependency can’t build a backlog that makes every later decision late too.

See Redis / Valkey: real-time decisions.

Sizing decisions made on day one

Some things are expensive to change later, so they are set generously from the start:

Decision	Why
Keyed topics start with 64 partitions (configurable with SP_KAFKA_PARTITIONS)	Adding partitions to a keyed topic later moves customers to different partitions and breaks per-customer ordering
Redis/Valkey sized per active customer, account and counterparty × buckets	Memory depends on how many of them are active and in how many ways rules slice their activity, not on transaction count. See Feature store: memory
Redis Cluster for high decision volumes and the design ceiling; Sentinel acceptable at lower volumes	One primary eventually becomes the bottleneck; keys are hash-tagged so Cluster shards share the load evenly

Local development uses fewer partitions

For a laptop, the example settings suggest 3–6 partitions and a replication factor of 1 (-1, the broker default). Do not copy these to production. On a production Kafka cluster use the default 64 partitions (or more) and a replication factor of 3.

Staying up when things fail

Safeguard	What it protects against
Several copies of each stateless role	One server failing
Kafka keeps events until they are processed	A component falling behind or restarting. The backlog waits on the conveyor
Dead-letter topic per component	One bad record blocking everything behind it
Replay from an earlier point	A bug fix that requires reprocessing. Hits and alerts are never duplicated; for the feature store, replays are safe for about a week back (see Stream evaluator: replays)
Duplicate protection on ingest	Retries counting the same money twice
Fail-open or fail-closed policy per institution	The feature store or rule engine being slow or down: every decision is still answered within the budget
A Redis spool for decided transactions	Kafka being down: decisions are still answered, and the transactions reach Kafka once it is back. See When Kafka is down
An outbox for webhooks, written in the same database transaction as the change	A receiver being down: events are retried for about 27 hours and never sent for a change that didn’t happen
Postgres high availability via Pigsty (Patroni for failover; PgBouncer for the app role’s connections where you need it), with read replicas for the web app and reports	Database server failure; heavy reporting slowing down operational work
Redis/Valkey Sentinel or Cluster	Feature-store server failure. Nothing of record is lost with it, but there is no command to refill it: see Feature store

Knowing it is working

• Prometheus metrics on a separate internal port: decisions by outcome, decision time, degraded decisions by cause, the Kafka spool’s depth, items received, transactions checked and batch times, notifications, webhook deliveries and persisted rows. See Monitoring. Pigsty ships a Prometheus and Grafana stack you can reuse.
• Structured logs (JSON in production) that log platforms can search.
• OpenTelemetry tracing, off until you point it at a collector. A transaction’s trace follows it from the API through Kafka, and each persister and stream batch links back to the transactions it handled. See Monitoring: tracing.
• A benchmark in the build pipeline that fails the build when a change makes a real-time decision slower. See The benchmark that guards decision speed. A benchmark against a reference cluster, for capacity, is still planned.

What to watch first:

1. Decision API p99 latency and the share of degraded decisions
2. Consumer lag per topic (how far behind each component is)
3. Messages arriving on dead-letter topics
4. Redis/Valkey memory use
5. Postgres replication lag

The benchmark that guards decision speed

SpeculaPro’s source repository carries a small, fixed benchmark of the real-time decision path, set up to run in the build pipeline (the bench job of the CI workflow) on every push. It answers one question: did this change make a decision slower? It is not a capacity test. How much load an installation can take is the job of a load test on your own hardware.

What it does:

• It runs the real decision service against a real Valkey and PostgreSQL, with the real-time rules of the fraud and gh-psp packs switched on, and sends a fixed, repeatable sequence of payments one at a time. With no queue, the time measured is the cost of one decision.

• It compares three things with a baseline kept in the repository, and fails the build when any is worse by more than its tolerance:

  Measure	Fails the build when it is worse by more than
  99th-percentile decision time	75%
  Median decision time	50%
  Redis/Valkey commands per decision	10%

• Decision times are not compared in milliseconds. Build machines are shared and their speed varies, so each run also times a fixed reference operation on the same machine, and the decision time is expressed as a multiple of it. The comparison uses the middle result of five runs. The command count doesn’t depend on the machine at all, which is why its tolerance is the tightest.

The tolerances are wide on purpose. The benchmark is there to catch a change that matters, such as a rule engine that became much slower or a feature read that turned into many round trips, not a drift of a few per cent.

The baseline is a laptop measurement, and the job has not yet run in CI

The committed baseline was measured on a developer’s laptop, not on the build machines the job runs on, and the job itself has not yet run in the build pipeline. The reference-operation method is meant to let a baseline carry from one machine to another, but that has not been shown on a build machine. The first run there should be read and its result committed as the baseline. Until then, treat a pass or a failure of this job as indicative.

The baseline is refreshed only in a change that alters what a decision costs on purpose, such as a real-time rule added to a pack. The steps are in the repository: the package comment of backend/internal/bench and assets/plan/04-benchmarks.md. Self-hosting operators don’t run this benchmark; contributors and anyone building SpeculaPro from source do.

Why it’s built this way

• Load testing from the start, because a monitoring system that falls over on salary day is worse than none.
• Horizontal scaling keeps costs proportional to volume and avoids a single giant server as a point of failure.
• The database is kept off the critical path, so real-time answers don’t depend on disk speed.

What this means for you

• Small self-hosters: one server running --role=all plus one each of Postgres, Redpanda and Valkey is enough to start. Add high availability before you go live.
• Large institutions: plan Kafka partitions and Redis memory up front, and split roles as soon as you see sustained load.
• Everyone: agree your availability expectations with us (SaaS) or your operations team (self-hosted), and alert on the five signals above.

Related

• Components and Kafka topics
• Feature store
• Self-hosting