Load testing

In short

SpeculaPro includes a load generator that sends realistic mobile money traffic to the ingest API as fast as you ask, or to the real-time decision API at a steady rate, then reports throughput and how long requests took. Use it to:

• fill a development or test installation with plausible data,
• check that a new installation can take your peak volumes before go-live,
• compare settings (number of ingest nodes, persist workers, database tuning).

Never point it at production

The generator creates thousands of fake customers, accounts and transactions in whichever workspace its API key belongs to. They can’t easily be removed afterwards. Use a dedicated test workspace on a test installation.

What it sends

Each transaction looks like everyday Ghanaian mobile money traffic:

• amounts mostly small (typically a few cedis to a few hundred), with a long tail, and about 1 in 100 transactions two hundred times larger;
• channels weighted towards mobile money (momo), plus agent, bank transfer, instant payment, card and USSD;
• counterparties on MTN MoMo, Telecel Cash and AT Money, with Ghanaian names and phone-style identifiers;
• spread over a configurable number of customers (50,000 by default), each with one wallet account, all in GHS.

Every run uses fresh external_ids (lg-<run>-<n>), so repeated runs add new transactions rather than producing duplicates.

How to run it

1. Create a test workspace (see Quickstart) and an API key with the ingest:write scope in it (see Getting an API key).

2. Make sure the target runs ingest and persist. make run locally runs both.

3. Run the generator from the backend directory:

   cd backend
   SP_LOADGEN_KEY=spk_… go run ./cmd/loadgen -url http://localhost:8080 -n 200000 -batch 500 -c 16

4. Read the result:

   sent 200000 txns in 5.123s → 39039 txn/s (accepted 200000, failed requests 0)
   request latency (batch of 500): p50 180ms  p95 320ms  p99 410ms  max 650ms

   (Example output. Your numbers depend entirely on your hardware and settings.)

Options

Flag	Default	Meaning
-url	http://localhost:8080	Base address of a server running the ingest role
-n	100000	Total transactions to send
-batch	500	Transactions per request. Keep it at or below SP_INGEST_MAX_BATCH (500 by default), or every request is refused
-c	8	Requests in flight at the same time
-customers	50000	Number of distinct customers (and wallet accounts) the traffic is spread over
-mode	ingest	ingest sends batches to the ingest API; decide asks for real-time decisions (see below)
-rate	1000	decide only: decisions requested per second
-duration	30s	decide only: how long to keep asking

The API key goes in the SP_LOADGEN_KEY environment variable, never on the command line, so it doesn’t end up in your shell history.

Reading the numbers

Output	Meaning
txn/s	Transactions the ingest API accepted per second, counted from successful requests. This is not the database write rate
accepted	Items the server reported as accepted. Lower than sent means some were rejected; check the server log
failed requests	Requests that errored or didn’t return HTTP 200 (whole batches lost). Should be 0
p50 / p95 / p99 / max	Time for one request of a whole batch, not per transaction

The generator measures the ingest side only. The ingest API answers once transactions are safely on Kafka; the persister writes them to PostgreSQL afterwards. To see whether storage keeps up, watch the persister’s consumer lag during and after the run. If lag keeps growing during the run and takes long to drain afterwards, the persister is your limit; add persist workers.

How to load-test real-time decisions

In decide mode the generator asks for one decision per request, at a fixed rate, whether or not earlier requests have been answered. That is how payments really arrive: your customers don’t wait for SpeculaPro to catch up. A server that falls behind shows it as rising latency, degraded answers or dropped requests, not as a politely lower request rate.

1. Use a test workspace with the rule packs and decision budget you intend to use in production, and an API key with the decide scope.

2. Make sure the target runs the decision role, plus persist and stream if you want the full cost: every decided transaction is also stored and monitored afterwards, and that work uses the same Redis/Valkey.

3. Run it, starting below your expected peak and raising -rate step by step:

   cd backend
   SP_LOADGEN_KEY=spk_… go run ./cmd/loadgen -mode decide -url http://localhost:8080 -rate 200 -duration 60s -c 512

4. Read the result. It has this shape (numbers left out; yours depend on your hardware, rules and data):

   target 200/s for 1m0s: … answered in … → … decisions/s; errors …, dropped (all 512 connections busy) …, degraded …
   client latency (from due time): p50 …  p95 …  p99 …  p99.9 …  max …
   server latency (latency_ms):    p50 …  p95 …  p99 …  p99.9 …  max …
   outcomes: map[allow:… block:… review:… step_up:…]

Output	Meaning
decisions/s	Decisions answered per second
errors	Requests that failed or didn’t return 200. Should be 0
dropped	Requests the generator couldn’t even send, because all -c connections were still waiting for answers. Anything above 0 means the server can’t keep up at this rate; raise -c only if you’re sure the client is the limit
degraded	Answers given by the workspace’s failure policy because the checks didn’t finish within the budget. This is the number to watch
client latency	Measured from when each request was due, so time spent queueing in the client counts
server latency	The latency_ms SpeculaPro reports in each answer: time spent deciding on the server
outcomes	How many of each answer. With random test traffic, most are allow

Your capacity is the highest rate at which degraded stays within what you agreed with the business (for example under 1%) and errors and dropped stay at 0. Watch Redis/Valkey CPU at the same time: it is usually the limit. See The decision role.

Test on hardware like production’s

Decision capacity depends heavily on Redis/Valkey’s CPU and on the network between decision nodes and Redis. A laptop running every component at once is a poor guide: the processes compete for the same cores. Run the test on production-like servers before you commit to volumes.

Reference measurements

These ingest and storage figures were measured by the SpeculaPro team on a single laptop: everything on one MacBook, in-memory Kafka, PostgreSQL 18 with no tuning. No decision figures are published yet: they will be, once measured on production-like hardware.

What	Result
Ingest API, accepted	≈ 44,000 transactions per second, 0 failures
Persister, 1 worker	≈ 7,800 rows per second
Persister, 8 workers	≈ 12,700 rows per second

Single-laptop measurements, not capacity claims

These numbers show the software works at volume on modest hardware. They are not a promise of what your installation will do. Real Kafka replicates every message to several brokers and real PostgreSQL has replicas and backups, both of which cost throughput, while real servers have far more cores and faster disks. Measure your own installation. The engineering targets are in Scaling and performance.

Tips for a meaningful test

• Test on hardware like production’s, with the same Kafka replication factor and PostgreSQL settings.
• Run long enough to see steady state: at least several minutes, or several million transactions.
• Raise -c until throughput stops increasing; that is the ingest side’s ceiling for your setup.
• Load FX rates first if you change the currency. The generator sends GHS only, so it needs none in a GHS workspace.
• Watch PostgreSQL and Kafka (CPU, disk, lag) at the same time. The slowest component sets the real limit.

Related

• Sending data
• Persister
• Running roles
• Scaling and performance