Risk models for operators

In short

A risk model gives each transaction an anomaly score that rules can read. For you as the operator:

• Nothing new to deploy. Models are trained, stored and served by the roles you already run, in PostgreSQL. There is no model server, no GPU and no extra data store.
• Training is the only heavy part. It runs in the worker role, in memory, like a backtest.
• Scoring needs no setting. A workspace with no shadow or live model is not scored at all.
• Retraining is a workspace’s choice. People train models in the app, and a workspace can switch on a weekly or monthly retraining schedule there, which the worker runs. Either way the result is a draft that people still have to promote.

People do the rest in the app: see Risk models: train, shadow, go live.

What runs where

Role	What it does for models
api	The Models page and the Risk models API: the registry, queuing a training, state changes, change requests
worker	Runs trainings, checks workspaces’ retraining schedules once a minute and queues the trainings that are due, and once an hour measures drift for every shadow and live model
decision	Scores each real-time decision with the workspace’s live model, in the same request as the rules
stream	Scores each transaction once with the live and the shadow model, feeds the live score to post-event rules, and counts what was scored for drift

The decision and stream roles load a model from PostgreSQL the first time a workspace needs it and keep it in memory. They notice a new shadow or live version when they next re-read the workspace’s rules: within SP_DECISION_STATE_TTL (10 seconds) for decisions and about 15 seconds for the stream. No restart is needed.

If a stored model can’t be loaded, the role logs models: a model could not be loaded; scoring without it and carries on as if the workspace had none: rules read model.available == false, so every rule guarded by model.available stops firing for that workspace on that process. Scoring never stops a decision or a transaction. The failure is counted in speculapro_model_load_failures_total, held in speculapro_models_not_loaded until the model loads or leaves service, and for a live model raises the SpeculaProLiveModelNotLoaded alert. The load is tried again each time the workspace’s rules are re-read.

Settings

All are read by the worker role. See also Configuration.

Setting	Default	Meaning
SP_MODELS_WORKERS	1	Trainings at once in this process. 0 turns training and drift measurement off in this process, leaving them to other workers
SP_MODELS_MIN_ROWS	5000	The fewest transactions a training window may hold. A training on less is refused with a clear message. At least 100
SP_MODELS_TIMEOUT	1h	A training that takes longer is stopped and marked failed. At least 1m
SP_MODELS_MAX_MEMORY_MB	1024	Feature state one training may hold in memory. At least 64
SP_MODELS_MAX_EVENTS	30000000	Transactions one training may replay, the history before the window included. At least 1000
SP_MODELS_DRIFT_DAYS	7	The last so many days are compared with the training data. 1 to 90
SP_MODELS_DRIFT_INTERVAL	1h	How often drift is measured. At least 1m

A value out of range stops start-up with config: SP_MODELS_* settings are out of range ….

Set SP_MODELS_MIN_ROWS and SP_MODELS_DRIFT_DAYS to the same value on the api role: the API checks the minimum before it queues a training, and the Models page shows the drift window.

Don’t lower the minimum to get a model sooner

A model fitted to a handful of transactions calls everything unusual. The default of 5,000 is already modest. Lower it only for a demonstration.

Across the installation at most two trainings run at once, and one per workspace. Others wait in the queue.

Memory and time

A training replays the workspace’s history for the chosen window through the same point-in-time engine as backtests, reading PostgreSQL, then fits the model on a random sample of at most 50,000 transactions (500,000 at the most a person can ask for).

• Memory grows with the number of customers, accounts and counterparties active in the window, not with the number of transactions. SP_MODELS_MAX_MEMORY_MB caps it; a training that would go over fails with a message telling the person to choose a shorter window or ask you to raise the setting.
• Keep the worker’s memory limit above SP_BACKTEST_WORKERS × SP_BACKTEST_MAX_MEMORY_MB + SP_MODELS_WORKERS × SP_MODELS_MAX_MEMORY_MB, plus what the worker needs for its other jobs. With the defaults that is 2 × 1024 + 1 × 1024 = 3 GB for backtests and trainings alone. The Helm chart’s default worker limit is 3 GiB, which leaves nothing spare for the worker’s other jobs: raise roles.worker.resources.limits.memory, or lower one of the settings, if backtests and trainings can run at the same time.
• Time grows with the transactions in the window and the history before it that fills the 30-day aggregates. There are no published timings yet; measure on your own data before you promise anyone a schedule.
• Database load is the same kind as a backtest’s: long sequential reads of the workspace’s transactions, under the workspace’s row-level security.

To keep trainings off a worker you reserve for notifications, set SP_MODELS_WORKERS=0 there and leave another worker on 1. Make sure at least one worker has it above 0, or nothing trains and drift is never measured.

A training whose worker dies is noticed after two minutes without a heartbeat and marked failed with The worker stopped during the training. Train again.

The command line

speculapro models list <workspace>
speculapro models train <workspace> [--days n] [--note text] [--seed n] [--wait]

<workspace> is the workspace’s slug. Both commands need the database settings and nothing else.

How to list a workspace’s models

# Docker Compose
docker compose run --rm cli models list acme

VERSION  STATE    TRAINING   WINDOW                        ROWS  DRIFT
3        shadow   succeeded  2026-07-14..2026-10-12      184220  stable (PSI 0.06)
2        live     succeeded  2026-04-10..2026-07-09      171904  moderate (PSI 0.14)
1        retired  succeeded  2026-01-05..2026-04-05      120377  -

How to queue a training

docker compose run --rm cli models train acme --days 90 --note "Monthly retrain"

queued: model 0192f3a1-7c1e-7b52-9a55-2f2f0f6f4d11. A worker will train it; it stays a draft until someone moves it to shadow.

Option	Default	Meaning
--days	90	Length of the training window, ending now. 7 to 180
--note	Trained from the command line	Kept with the version
--seed	Random	Give a version’s seed to train the same model again
--wait	Off	Train in this process and wait, instead of leaving it to a worker. The process then needs the memory a training needs

The same rules apply as in the app: the minimum number of transactions, and one training at a time per workspace. A refusal is printed and the command exits with an error.

A version trained from the command line has no “Trained by”: its audit entry is recorded as the system. It is a draft. The command line can’t move a model to shadow, propose it or approve it. Those are for people, in the app, and going live always needs two of them.

How to schedule retraining

You don’t need a scheduler of your own. A workspace switches on a retraining schedule on its Models page, weekly or monthly, and the worker queues the trainings. See The retraining schedule.

What the worker does:

• Once a minute it looks for schedules that are due and queues each training through the same code as the API. The training then waits its turn like any other, under the same SP_MODELS_* limits: the minimum number of transactions, one training per workspace at a time, the number running at once, the timeout and the memory budget.
• Several workers queue a due training once. Each schedule is locked while it is handled.
• Another training in the way defers the occasion for an hour. Too little data skips it until the next occasion, with the reason shown on the workspace’s Models page.
• If the training can’t be queued for another reason, such as the database, the schedule stays due, the worker logs models: queue scheduled training; it stays due and tries again on its next pass.
• A worker that was away for several periods queues one training, not one for each it missed.
• Workspaces that are not active, or are in a move, are left alone. Their schedule stays due until they are active again.
• It only queues. Each occasion leaves one more draft in the registry. Somebody still has to move it to shadow, watch it and propose it, and somebody else approve it. Drafts are never deleted.

A queued training logs models: scheduled training queued at INFO, with the workspace, the version and the next occasion. Without a running worker, nothing is queued and schedules stay due.

You can still queue a training from your own scheduler with models train, for example at an hour you choose, since a workspace’s schedule can’t be set to a time of day. A run is refused while a training is already queued or running for that workspace, and when the window holds too few transactions, so alert on the job’s exit status.

What to watch

Metrics

Metric	Type	Labels	Meaning
speculapro_model_scores_total	Counter	state (live, shadow)	Transactions scored by a risk model
speculapro_model_trainings_total	Counter	result (succeeded, failed, refused, cancelled)	Trainings by result. refused is too little data or too many transactions to replay: a person’s choice of window, not a fault
speculapro_model_scheduled_trainings_total	Counter	result (queued, skipped, deferred, error)	Occasions of workspaces’ retraining schedules. skipped: too little data, tried at the next occasion. deferred: another training was in the way, tried again in an hour. error: it could not be queued and stays due
speculapro_model_drift_psi	Gauge	state (live, shadow), of (score, feature)	The highest population stability index among the installation’s models, for the score and for the most shifted feature. Under 0.1 is stable; above 0.25 is a significant shift
speculapro_models_drifting	Gauge	state	How many models have drifted significantly
speculapro_model_load_failures_total	Counter	state (live, shadow)	Attempts to load a model in service that failed. The workspace is scored without it until one succeeds
speculapro_models_not_loaded	Gauge	state	Models in service that this process could not load when it last tried. A live one means rules guarded by model.available are not firing for that workspace

No workspace is ever a label. The drift gauges carry the worst case across the installation; the worker log and the workspace’s Models page say which workspace and version. Models with fewer than 500 transactions scored in the window are left out of the gauges.

Alerts

The Helm chart’s PrometheusRule carries three. On other platforms, build the same from the expressions.

Alert	Fires when	Severity	What to do
SpeculaProModelDrift	max(speculapro_model_drift_psi{state="live"}) above prometheusRule.thresholds.modelDriftPsi (0.25) for 6 hours	warning	A live model’s traffic no longer looks like its training data, so rules that read the score fire differently from when the model was approved. Find the workspace in the worker log (models: significant drift) and tell its compliance team to train a new version, run it in shadow and promote it
SpeculaProModelTrainingFailing	Any training ended failed in the last 6 hours (refused and cancelled don’t count)	warning	The Models page shows the message; the worker log (models: training failed) has the cause. Usually the worker’s memory limit against SP_MODELS_MAX_MEMORY_MB, or SP_MODELS_TIMEOUT
SpeculaProLiveModelNotLoaded	max(speculapro_models_not_loaded{state="live"}) above zero for 2 minutes	critical	A pod can’t load a workspace’s live model, so every rule guarded by model.available has stopped firing there. The decision and stream logs (models: a model could not be loaded) name the workspace, the version and the cause, usually a damaged or missing stored model or one from a newer release. Retire the version or promote another on the workspace’s Models page

Only live models raise this monitoring alert. Separately, the worker notifies the workspace’s own people (those with models.manage or models.approve) when a shadow or live version’s drift is significant, once a week for each version while it lasts; see Notifications.

Logs

Message	Level	Meaning
model training runner started	INFO	The worker will train and measure drift
model training disabled in this process (SP_MODELS_WORKERS=0)	INFO	This worker does neither
model trained	INFO	With the rows read, the sample size and the stored model’s size
model training ended	INFO	Cancelled, refused or failed, with the sentence the person sees
models: training failed	ERROR	A failure that isn’t the window’s fault. The cause is here
models: training interrupted, marked failed	WARN	A worker died mid-training
models: significant drift	WARN	With the workspace, version, state and PSI
models: model loaded	INFO	A decision or stream process picked a version up
models: a model could not be loaded; scoring without it	ERROR	The workspace is being served as if it had no model, with tenant, model, version, state and err. Counted in speculapro_model_load_failures_total; a live one raises SpeculaProLiveModelNotLoaded

What is stored, and where

Everything is in PostgreSQL, under row-level security per workspace.

Table	Holds
risk_models	The registry: one row per version with its state, training window, parameters, seed, feature list, metrics, model card, drift, who did what, and the stored model itself (artifact, gzip-compressed, with its SHA-256)
risk_model_stats	What each shadow or live model scored, per day: a count and histograms of the score and of each feature. No transaction and no customer
rule_hits.model, decisions.model	The live model’s score for the transaction behind a hit or a decision, with its reasons. Shadow scores are never written here

Backups

There is nothing extra to back up: a PostgreSQL backup holds every model. After a restore, the decision and stream roles load the live and shadow models again by themselves.

A model can also be rebuilt: the same window, features, parameters and seed give the same stored model, byte for byte, as long as the transactions of that window are unchanged. Compare the SHA-256 on the model card.

The stored model and erasure

Your data protection officer will ask what a model remembers about a person. The stored model holds:

• for each tree, which feature each split tests, the threshold it compares with (a number chosen at random between the smallest and largest value in that part of the sample), and how many training transactions reached each node;
• for each feature, its smallest, middle and largest value in the training sample and nine boundary values between them, used to measure drift;
• the spread of the model’s scores on its training sample.

It holds no transaction, no customer, account or counterparty identifier, no name and no contact detail. It is numbers about the whole sample. When a customer is later purged, erased or pseudonymised, there is nothing of theirs in the model to find or remove, and SpeculaPro doesn’t retrain or alter models when that happens.

Two things are worth saying plainly to whoever asks:

• Single extreme values are kept as numbers. A feature’s largest value, such as the largest amount seen in the window, is one transaction’s value. It is stored without anything that says whose it was.
• Scores on hits and decisions are a separate matter. The score and reasons saved with a rule hit or a decision describe one transaction (for example 14 times the customer’s 30-day average). They are deleted with that hit or decision, under the retention period of the record they belong to. A decision that is pseudonymised instead of deleted keeps the model’s version, score and percentile and loses the model’s reasons, along with every rule’s explanation and feature values on it (decisions.reasons and decisions.shadow). Rule hits on alerts still quote feature values until the alert’s own retention period removes them. See Retention and legal holds.

Deleting a workspace deletes its models and their statistics with it.

Upgrading

Migration 00045_models adds the two tables, the model column on rule_hits and decisions, and the models.view, models.manage and models.approve permissions on the built-in roles. Nothing is trained by the upgrade, and no workspace is scored until someone moves a model to shadow. Custom roles are not changed.

Troubleshooting

Symptom	Likely cause	What to do
A version stays Waiting to train	No worker has SP_MODELS_WORKERS above 0, or two trainings are already running on the installation	Check the worker log for model training runner started; wait for the others
Training failed: …more feature memory than one training may use	The window has too many active customers and accounts for SP_MODELS_MAX_MEMORY_MB	A shorter window, or raise the setting and the worker’s memory limit together
Training failed: The training took longer than the 1h limit	A very large window	A shorter window, or raise SP_MODELS_TIMEOUT
The worker is killed during a training	Its memory limit is below backtests plus trainings	Raise the limit as in Memory and time
A model is Shadow or Live but Activity is empty	The stream role isn’t running, or is behind	Check the stream role and its lag
Drift says Too little data for days	Fewer than 500 transactions scored in SP_MODELS_DRIFT_DAYS	Normal for a quiet workspace. Nothing alerts
Drift in the registry never changes	No worker measures it (SP_MODELS_WORKERS=0 everywhere)	Set it above 0 on one worker

Related

• Risk models, the concept
• Risk models: train, shadow, go live
• Risk models API
• Configuration
• Monitoring
• Worker