A real-time decision, step by step

In short

When a customer taps Send in a payment app, the payment provider can ask SpeculaPro “is this safe?” before the money moves. SpeculaPro answers allow, step_up, review or block, with the reasons, inside a time budget the institution sets. The customer never notices the check.

To be that fast, SpeculaPro answers from memory and Redis. It never writes to the database while deciding, and reads it only when something isn’t cached yet. Everything is recorded a moment later, in the background.

For what the answers mean and how to integrate, see Real-time decisions and Calling the decision API.

The journey of one payment

Ama sends GHS 2,400 from her mobile money wallet to a number she has never paid before, from a phone she started using an hour ago.

 Payment app          SpeculaPro decision role                             Background
     │                         │
 1   │── POST /decisions ────▶ │ check the API key; replay check (Redis)
 2   │                         │ validate, convert to base currency (cached rates)
 3   │                         │ load Ama, her wallet, the recipient   (in-process cache)
 4   │                         │ one Redis round trip: count this payment,
     │                         │   read velocity features and device / payee signals
 5   │                         │ run every live real-time rule  (compiled CEL)
 6   │                         │ combine into one answer
 7   │ ◀── answer + reasons ───│
     │                         ├── publish to Kafka ─────────────────────▶  8  persister stores the
     │                         │   (Redis spool if Kafka is down)              transaction and decision;
     │                         └── store replay record and statistics          post-event rules run;
     │                                                                          fraud alerts; webhooks

Step	What happens	Why it is fast
1. Authenticate, check for a replay	The API key must carry the `decide` scope. In parallel with validation, one Redis call asks whether this `external_id` was already decided and, if not, claims it for 30 seconds. A stored answer is returned with `replayed: true`; a request that finds the ID claimed by another still being decided gets `409` and retries	Keys are cached briefly on each node; the replay check runs while the request is validated
2. Validate and convert	The same checks as the ingest API. The amount is converted to the workspace’s base currency	Exchange rates are cached
3. Load the entities	Ama’s customer profile (type, status, risk rating, segment, KYC tier, designations), her wallet and the recipient. A recipient with no stored sanctions screening result (the first payment to them) has their name screened here against the workspace’s lists: see A counterparty seen for the first time	An in-process cache, 30 seconds by default. Only a cache miss reads Postgres. The screening is a search of a list index the process holds in memory: no query and no network call, and a name already searched is remembered. A match is written for review after the answer
4. Count and read features	This payment is added to Ama’s running counters in the same round trip that reads her features (“sends in the last 10 minutes”) and signals (“first time on this device?”, “first time paying this number?”). So a burst of payments is seen as it happens	One pipelined call to the feature store
5. Evaluate rules	Every real-time rule in its live slot runs, and shadow versions alongside them	Rules are compiled once and refreshed in the background about every 10 seconds
6. Combine	The strongest action wins; scores combine; the workspace’s thresholds may raise the answer. See how the answer is worked out	Pure in-memory computation
7. Answer	`block`, score 85, reason Payout soon after a SIM swap
8. Record	The transaction, with its decision, goes to Kafka, with the status `blocked` if the answer was `block`. From there it is stored, run through the post-event rules, turned into fraud alerts and sent to webhooks. The replay record and the Decisions page statistics are written to Redis in batches	Nothing here delays the answer

Steps 3 to 5 run against the time budget. Steps 1, 2 and 8 don’t wait on anything slow: the replay check, cached rates and an asynchronous publish.

The latency budget

The time budget is set per workspace under Settings → Workspace (150 ms by default, 20 to 2,000 ms). It is the most SpeculaPro will spend on steps 3 to 5 before the failure policy answers instead.

The engineering targets for a decision, measured on the server, are:

Target	Value
Median (p50)	10 ms or less
99th percentile (p99)	50 ms or less

These are targets, not measured guarantees: they will be confirmed on production-sized hardware during a pilot. The Decisions page and the speculapro_decision_duration_seconds metric show what your installation actually achieves. See Scaling and performance.

When something is wrong

The checks can’t finish: fail open or fail closed

If steps 3 to 5 don’t finish inside the budget, the workspace’s failure policy answers and the response says degraded: true:

Cause	When
`timeout`	The checks ran past the budget
`dependency`	Redis, or Postgres on a cache miss, failed, or the checks crashed. (A single rule that errors is skipped and logged; it doesn’t degrade the decision)
`overload`	The node was already evaluating as many decisions as it allows at once (4,096 by default); new ones are answered at once rather than queued

Policy	Answer	Afterwards
Fail open	`allow`	The payment is still recorded and monitored post-event, and a fraud alert asks someone to review it
Fail closed	`review`	The payment is still recorded and monitored post-event. Your own systems hold it, and a fraud alert asks someone to review it and release or decline it there

Worked example. A mobile money operator processes salary payments on the 25th. At 09:00 the Redis cluster that holds the running totals fails over, taking 20 seconds.

With fail open, the payments in those 20 seconds are allowed. Each raises a fraud alert for review, and the post-event rules check them once Redis is back. A suspicious one is found minutes later, but the money may already have moved.
With fail closed, those payments are answered review. None moves unchecked, but every customer paying in those 20 seconds sees “payment pending”, and your operations team has a queue to clear: each held payment raises a fraud alert, so the queue is in SpeculaPro’s alert queue too.

The request can’t be handled at all

A few failures can’t be answered by the policy, because the transaction couldn’t even be built: the workspace’s settings couldn’t be loaded for the first time, or exchange rates couldn’t be looked up. Then the API answers 503 decision service unavailable; apply your fallback and retry, and the payment system applies its own fallback.

Kafka is down

The answer never waits for Kafka. The producer buffers and retries on its own; if delivery finally fails, the event is parked in a Redis list (the spool) and replayed every second once Kafka is back. Replays are safe: the transaction’s ID is derived from its external_id, so nothing is stored twice.

A clean stop (SIGTERM, a rolling restart) loses nothing either. Once the last request has been answered, the decision node gives Kafka up to 10 seconds to take every buffered event and parks any it couldn’t deliver in the spool. It then writes the remaining replay records, which let a retry get the original answer, and closes its Redis connection only after that.

Only if Kafka and Redis both fail is an event lost. That is logged as an error and counted in speculapro_decision_events_lost_total; alert on any increase. See Monitoring.

Why it’s built this way

No database writes on the hot path. Waiting for a disk write on every payment would blow the budget under load. The decision is written a moment later, from Kafka.
Count while reading. Recording the payment in the same Redis round trip that reads the features means a fraudster firing ten payments in a second is seen on the second payment, not after a background job catches up.
Precompiled rules, refreshed in the background. A request never waits for rules to load, except the very first one for a workspace.
Evidence is never skipped. Writing happens after the answer, not instead of it. Every decision is recorded with the rule versions evaluated and the feature values seen, including decisions made while degraded.

What this means for you

Product teams: design what your app does on step_up, review and block before you go live, and set your own timeout above the workspace’s time budget.
Risk: set the failure policy deliberately.
Compliance: every decision, including those made during an outage, is recorded and can be explained afterwards on the Decisions page.
Operations: watch the degraded and lost-event metrics.