Feature store

In short

Many rules ask about the past: “How much cash has this account received in the last 7 days?”, “How many different people sent money to this wallet today?”.

Adding up a week of transactions every time a payment arrives would be far too slow. So SpeculaPro keeps running tallies that are updated as each transaction goes past, like a shop that updates its daily takings with every sale instead of recounting the till each time. A rule asks for a tally by name; these named tallies are called features, and the place they are kept is the feature store.

The feature store lives in Redis or Valkey. Three parts of SpeculaPro write to it: the stream evaluator, for every transaction and account event; the real-time decision service, for every payment it decides; and the ingest API, which applies account events the moment they arrive so a decision a second later sees them.

Tallies start with your first transaction

Every transaction the stream evaluator receives updates the hour and day tallies, whether or not any rule is enabled yet. So a rule enabled 30 days after you started sending data already has 30 days of history. Windows of 2 hours or less are the exception: see Minute and 5-minute buckets only where a rule needs them. Transactions sent before the stream evaluator was running, or loaded straight into Postgres, are not counted, and nothing goes back to count them later.

Who has tallies

Every transaction updates the tallies of up to four entities:

Entity	Identified by	Updated when
account	Your account reference (account_external_id)	The transaction, or an account event, names an account
customer	Your customer reference (customer_external_id)	Always, and for every account event
counterparty	The counterparty’s fingerprint (its normalised identifier, provider and name)	The transaction has a counterparty
device	A keyed hash of the device_fingerprint	The transaction, or an account event, carries a fingerprint

A rule chooses which entity to ask about. “Cash-in to this account in 7 days” and “cash-in to this customer, across all their accounts, in 7 days” are different features.

What a feature can measure

A feature is defined by the rule that uses it: which entity, which measurement, which transactions, over which window. The full syntax is in the features reference.

Measurement	Meaning	Unit
count	How many transactions	Number
sum	Their total value	Base currency, major units (cedis, not pesewas)
max	The largest single transaction	Base currency, major units
avg	Total value ÷ count	Base currency, major units
distinct_counterparties	How many different other parties	Number (an estimate, see below)
distinct_customers	On a device: how many different customers used it. On a counterparty: the same as distinct_counterparties	Number (an estimate)

Each measurement can be narrowed to:

• a direction: money in, out, internal, or any (the default), or event to count account events instead of money, and
• either one transaction type (for example cash_in) or one channel (for example momo), not both. With event, the type is the event type, such as otp_failed.

Distinct counts can only be narrowed by direction. Account events can only be counted, and are kept apart from money: any never includes them.

Windows run from 1 minute to 180 days (1m to 4320h). All amounts are in your workspace’s base currency, after conversion, so a rule’s threshold works whatever currency the payment was made in.

Who counts as “the other party”

• For an account or a customer, the other party is the transaction’s counterparty. distinct_counterparties = how many different people or businesses this account (or customer) paid or was paid by. Transactions without a counterparty are not part of this count.
• For a device, the other party is the customer. So a distinct count on a device = how many of your customers used it, through transactions or account events.
• For a counterparty, the other party is your account (or your customer, when the transaction names no account). So distinct_counterparties on a counterparty = how many of your accounts paid it, or were paid by it. That is exactly the question behind fan-in rules: “How many of our customers are sending money to this one outside wallet?”

The counts include the current transaction

When a transaction is evaluated, its own contribution is added first, then the features are read. A rule that asks “5 or more cash-ins in 24 hours?” fires on the fifth cash-in, not the sixth.

distinct_counterparties is an estimate

Distinct counts use HyperLogLog, a standard technique that counts unique items in a few kilobytes instead of remembering every one. The answer is very close but not exact: typically within about 1%, so for small numbers (under a few hundred) it is usually exact. Don’t write a rule that depends on the difference between 1,000 and 1,005 distinct counterparties.

Time buckets and exact windows

Tallies are kept in buckets, each covering a slice of time:

Bucket	Used for windows	Kept for
1 minute	Up to 15 minutes	About 20 minutes
5 minutes	16 minutes to 2 hours	About 2 hours 10 minutes
1 hour	Over 2 hours, up to 48 hours	About 2 days
1 day	Longer than 48 hours	About 182 days
10 days	Longer than 48 hours (a shortcut: ten day-buckets in one)	About 192 days

A window is answered by adding up the buckets that cover it. That is fast whatever the volume, because a 180-day window reads about 36 buckets instead of 180, and memory stays bounded because old buckets expire on their own.

The price is that windows are rounded to whole buckets: the oldest, partly covered bucket is left out.

Minute and 5-minute buckets only where a rule needs them

At high volume the fine buckets are most of the work, so a transaction writes a counter into them only if some running rule (live or shadow, real-time or post-event) reads that counter with a window of 2 hours or less. Hour and day buckets always get every counter, so a rule switched on later finds its history; a new short-window rule starts from zero and is complete after one window length. Account events always go into the fine buckets: they are few, and short-window takeover rules depend on them.

Late data misses expired buckets

A bucket is deleted a fixed time after it ends. A transaction or event that arrives after its bucket has gone isn’t counted in it: more than about 20 minutes late, it is missing from windows of 15 minutes or less; more than about 2 hours 10 minutes late, from windows of 2 hours or less; more than about 50 hours late, from windows of 48 hours or less. It still counts in longer windows. Live traffic isn’t affected, because for a transaction made now, data that old is outside those windows anyway; a backlog sent late is. See Late and backdated data.

Windows are slightly shorter than they say

• Up to 15 minutes: whole minutes, so between window − 1 minute and window.
• Up to 2 hours: whole 5-minute blocks, so between window − 5 minutes and window.
• Up to 48 hours: whole hours. The window starts at the top of the hour after “now minus the window”. It covers between window − 1 hour and window.
• Longer than 48 hours: whole days. The window starts at midnight after “now minus the window”. It covers between window − 1 day and window.

Hours and days are UTC. For Ghana (UTC+0) that is the same as local time. For workspaces in other time zones, day boundaries fall at UTC midnight, for example 01:00 in Lagos.

Worked example: a 24-hour window

A cash-in arrives on Tuesday at 14:35. A rule asks for the sum of cash-in over 24h.

• “Now minus 24 hours” is Monday 14:35, inside the Monday 14:00–15:00 bucket.
• That partial bucket is left out; the window starts at Monday 15:00.
• The window covers Monday 15:00 to Tuesday 14:35: 23 hours 35 minutes, including the current transaction.

A cash-in on Monday at 14:50 is not counted. One on Monday at 15:05 is.

Worked example: a 7-day window

The same transaction, Tuesday 14:35. A rule asks for the count of transfers out over 168h (7 days).

• “Now minus 7 days” is the previous Tuesday 14:35, inside that Tuesday’s day bucket.
• That partial day is left out; the window starts at Wednesday 00:00.
• The window covers Wednesday 00:00 last week to Tuesday 14:35 today: six full days plus today so far.

So a 7-day window always looks back between 6 and 7 days. If your policy says “seven full days”, ask for 192h (8 days): that covers between 7 and 8 days.

count of transfers out, account +233•••••••89, 168h window evaluated Tue 14:35 (UTC)

  Tue      Wed   Thu   Fri   Sat   Sun   Mon   Tue
 (dropped)  2     0     3     1     0     4    2 so far   ── count ──▶ 12

Customer profiles

Next to the tallies, each customer has a small profile used by the real-time signals in the CEL reference:

Kept	Used for
When each device, counterparty, country and IP network (/24 or /48) was first seen in their transactions	device.is_new, counterparty.is_new_for_customer, txn.country_is_new, txn.ip_is_new and the “hours since first seen” fields
Their latest location, and the one before it	txn.km_from_last, txn.kmh_from_last, txn.minutes_since_last_location
When each type of account event last happened	customer.hours_since

A profile lives for 180 days after the customer’s last activity. It is read and updated in the same step as the transaction, and remembers which transaction set each value, so a transaction checked twice (in a real-time decision, then post-event) gets the same answers both times.

Correct under retries and replays

• Each transaction is counted once per entity. SpeculaPro remembers, for the replay window (30 minutes by default, set with SP_STREAM_REPLAY_TTL), which transactions each entity has already counted. A transaction delivered twice (a retry, a restart, a rebalance) is not counted twice.
• Features are calculated as of each transaction. A batch of transactions is applied to the store in order, in one round trip, so each transaction sees exactly the transactions before it and none after. Within one account, order is strict; see Components: topics for how customer-level totals across several accounts behave.
• Values are snapshotted. The feature values each transaction was evaluated with are kept for the same replay window. If the transaction is processed again in that time, for example because the evaluator restarted before confirming its batch, the snapshot is reused, so the same hits come out, even though later transactions have since changed the tallies.

Replays older than the replay window count twice

Once the replay window has passed, SpeculaPro no longer remembers that it counted a transaction. If older transactions are processed again (for example after a stream outage longer than the window, or because the stream evaluator’s consumer group was renamed or reset), they are added to the tallies a second time, and long-window rules see inflated totals. Hits and alerts are still never duplicated. See Stream evaluator: replays.

Memory

Memory grows with the number of active entities and the number of ways their transactions are sliced, not directly with the number of transactions.

What lives in Redis/Valkey:

Data	One per	Kept for
Counters (count, sum, max)	Entity × bucket, with a few numbers for each direction, transaction type and channel seen	2 days (hourly) to 192 days (10-day)
HyperLogLog sets for distinct counts	Entity × bucket × direction, when a counterparty is known	Same as the bucket
“Already counted” markers	Entity × transaction or account event	Replay window (30 minutes by default)
Customer profiles	Customer: first-seen times per device, counterparty, country and IP network, two locations, and the latest time of each account event type	180 days after last activity
Feature snapshots	Transaction (one entry per feature the rules use)	Replay window

The last two grow with throughput × replay window: at 5,000 transactions per second and the default 30 minutes, about 9 million transactions are remembered at any time. A longer window survives longer outages but costs proportionally more memory. Measure memory with your own volumes before sizing production; see Load testing.

All keys of one entity share a Redis hash tag, so every update touches a single slot. The store therefore works the same on standalone Redis/Valkey, with Sentinel, and on Cluster.

Not the record, but not trivially rebuilt

The feature store holds only derived numbers. The real record of every transaction is in Postgres. Nothing of record is lost if Redis/Valkey is lost.

But there is no rebuild command for a running installation. The one rebuild that exists is inside a workspace move: the target region replays the workspace’s last 192 days of stored transactions and account events into its own Redis/Valkey before serving it. If the feature store’s data is lost otherwise:

• Tallies start again from empty. Rules with long windows under-count until their windows have refilled with new transactions (up to 180 days).
• Snapshots are lost too, so reprocessing old transactions would evaluate them against today’s tallies.

Refilling by replaying txn.ingested from Kafka into an empty store is possible in principle, but only for what Kafka still retains, and it re-runs today’s rules over those transactions. Treat Redis/Valkey as important: run it with replicas and failover (Sentinel or Cluster), and with persistence (AOF) enabled, so a restart does not empty it.

Layout	Suitable for
Standalone (one server)	Development and very small installations
Sentinel (one primary, replicas, automatic failover)	Most production installations below the v1 performance target
Cluster (data spread across several servers)	Very high volumes, towards the design ceiling

See Redis / Valkey for setup and Scaling and performance for sizing.

Still to come

• Locations and devices from account events, so a login can move the customer’s location or make a device known. Today account events are counted on the customer, the account and the device, and the time of the last event of each type is remembered; only transactions set the last location and make a device known.

Two things once listed here are built. “Unusual for this customer” rules read the customer’s own history through the avg aggregation and the same_hour and all_hours filters (see the features reference). Network patterns such as circular flows and fan-in followed by fan-out are found by scheduled rules, which query stored transactions on a timetable instead of reading running tallies.

What this means for you

• Rule authors: remember that windows round down to whole minutes (≤ 15 min), 5-minute blocks (≤ 2 h), hours (≤ 48 h) or whole UTC days (longer), and that the count includes the current transaction. Pick windows and thresholds with that in mind. See the features reference.
• Operations: monitor Redis/Valkey memory, run it highly available with persistence, and avoid replaying the stream evaluator over more than a week of history.
• Compliance: every hit records the exact feature values it was based on, so an alert can always be explained, even after the tallies have moved on.

Related

• Stream evaluator
• Account events
• Features reference
• Rule engine
• Redis / Valkey