Monitoring

In short

SpeculaPro gives you four ways to see whether it is healthy:

• Health checks, for your load balancer and uptime monitoring.
• Prometheus metrics, on a separate internal port, for dashboards and alerts on real-time decisions, webhooks and the persister.
• Structured logs, for your log platform.
• Traces (OpenTelemetry), when you switch them on, to follow one request through SpeculaPro and see where the time goes.

Monitor PostgreSQL, Kafka and Redis/Valkey with their own tools.

Health checks

Endpoint	Answers	Response	Use it for
GET /healthz	Is the process running and answering HTTP?	204 No Content	Liveness probes; load balancer checks
GET /readyz?role=…	Can it reach the dependencies these roles can’t work without? (api, decision, all: PostgreSQL; ingest: PostgreSQL and Kafka)	204 No Content, or 503 with not ready	Readiness probes. Give the roles the process serves, for example /readyz?role=api or /readyz?role=decision
GET /readyz	Without role: on a process that answers decisions (decision alone, or with api and ingest, including all), PostgreSQL only, like ?role=all, so it stays ready with Kafka or Redis down. On any other process, what its HTTP roles need, combined (for example ingest alone: PostgreSQL and Kafka)	204 No Content, or 503 with not ready	Safe as a load balancer’s check for any process. Give role explicitly when you want a different check. An unknown role gives 400
GET /api/v1/system/health	Can it reach its dependencies (PostgreSQL, Redis/Valkey, Kafka)?	200 with {"status":"ok"}, or 503 with {"status":"degraded"}	Uptime monitoring and alerting
GET /api/v1/system/version	Which build is running?	{"version":"…","commit":"…"}	Confirming an upgrade. Requires a signed-in user

/healthz deliberately checks nothing except the process itself, so a database outage doesn’t make your orchestrator restart healthy SpeculaPro processes in a loop.

/api/v1/system/health checks each dependency with a 2-second time limit:

curl -s https://speculapro.example.com/api/v1/system/health

{"status":"ok"}

{"status":"degraded"}

A degraded installation answers with HTTP 503 Service Unavailable, so any standard uptime monitor that alerts on non-2xx responses works without extra configuration.

Which dependencies each role checks

Readiness checks only what a role can’t work without, so an outage of one dependency doesn’t take out processes that can carry on:

Role	Checked by	Ready when	Why not more
api	/readyz?role=api on the HTTP port	PostgreSQL answers	People keep working cases during a Kafka or Redis outage; only parts such as live counters and rate limits degrade
decision	/readyz?role=decision on the HTTP port	PostgreSQL answers	Decisions keep answering during Kafka and Redis outages (failing open and parking events in the spool), so decision nodes must stay in the load balancer
ingest	/readyz?role=ingest on the HTTP port	PostgreSQL and Kafka answer	Without Kafka every item would be answered retry
all (everything in one process)	/readyz?role=all on the HTTP port	PostgreSQL answers	The process serves decisions, which must not lose fail-open
persist	/readyz on the metrics port	PostgreSQL and Kafka answer
stream	/readyz on the metrics port	PostgreSQL, Kafka and Redis answer
worker	/readyz on the metrics port	PostgreSQL answers
connector	/readyz on the metrics port	PostgreSQL and Kafka answer
analytics (optional)	/readyz on the metrics port	PostgreSQL, Kafka and ClickHouse answer	It copies both into the analytics store. api and worker only read the store and fall back to PostgreSQL, so ClickHouse is never one of their checks

A process that runs only background roles (persist, stream, worker, connector, analytics) has no HTTP port: its metrics listener answers /healthz (the process is up) and /readyz (the dependencies above). If such a process also runs decision, its metrics-port /readyz checks PostgreSQL only, for the same fail-open reason. When a check fails, the process logs dependency unhealthy with the dependency’s name, and the metrics-port /readyz answers 503 not ready: kafka (naming what failed). See Running roles: health checks.

Which dependency failed?

The list of failed dependencies is shown only to a signed-in user holding the settings.manage permission (by default, the Administrator role), so anonymous callers can’t map your infrastructure:

{"status":"degraded","failed":["redis"]}

For unattended monitoring, use the logs instead: every failed check writes a warning naming the dependency (see below).

Prometheus metrics

Every SpeculaPro process serves metrics in the Prometheus format at /metrics on its own listener, SP_METRICS_ADDR (default :9090). It is separate from the web app and API port on purpose, so the figures never reach the public network.

curl -s http://10.0.1.15:9090/metrics | grep '^speculapro_'

Setting	Default	Meaning
SP_METRICS_ADDR	:9090	Address of the metrics listener. off (or empty) switches it off

Keep the metrics port internal

The metrics endpoint has no sign-in. It contains no customer data, but it does show your volumes and decision outcomes. Bind it to an internal interface (for example SP_METRICS_ADDR=10.0.1.15:9090) or block the port at the firewall, and let only your Prometheus server reach it.

If the listener can’t start, for example because another process already uses the port, SpeculaPro logs metrics listener failed; stopping and the whole process stops. It never runs unobserved. Two SpeculaPro processes on one machine need two different ports.

To collect the metrics, add every SpeculaPro process to your Prometheus configuration:

scrape_configs:
  - job_name: speculapro
    scrape_interval: 15s
    static_configs:
      - targets: ["decision-1.internal:9090", "decision-2.internal:9090", "worker-1.internal:9090"]

On Kubernetes, use your usual service discovery or a ServiceMonitor that points at port 9090.

What is measured

All names start with speculapro_. Counters (ending _total) only go up; use rate() or increase() on them.

Real-time decisions (processes running the decision role):

Metric	Type	Labels	What it counts
speculapro_decisions_total	Counter	outcome (allow, step_up, review, block), degraded (true, false)	Decisions answered. degraded="true" means the workspace’s failure policy answered because the checks couldn’t finish
speculapro_decision_duration_seconds	Histogram		Time to decide, on the server, from receiving the request to having the answer
speculapro_decision_degraded_total	Counter	cause (timeout, dependency, overload)	Degraded decisions by why: the budget ran out, a dependency such as Redis failed, or the process already had SP_DECISION_MAX_INFLIGHT decisions in progress
speculapro_decision_replays_total	Counter		Retries answered with the original decision (same external_id, same request). Not counted in speculapro_decisions_total
speculapro_decision_events_spooled_total	Counter		Decided transactions parked in Redis because Kafka couldn’t take them. See Kafka: when Kafka is down
speculapro_decision_events_unspooled_total	Counter		Parked transactions delivered to Kafka later
speculapro_decision_events_lost_total	Counter		Decided transactions that reached neither Kafka nor the Redis spool, or that were found unreadable in the spool and dropped. The caller got its answer, but the transaction won’t be stored, monitored or sent as a webhook. Alert on any increase
speculapro_decision_spool_events	Gauge		Decided transactions waiting in the Redis spool for Kafka right now, sampled every second. Zero in normal operation. Every decision node reads the same spool, so they all report the same figure: use max, not sum
speculapro_decision_bookkeeping_overflow_total	Counter		Decisions whose replay record and statistics were written one by one because the batch queue was full. A steady rise means the decision node is overloaded
speculapro_decision_sanctions_set_parties	Gauge		Customers and counterparties with a confirmed sanctions match that this process holds in memory, across workspaces, so that a degraded decision still blocks them. At most 100,000 per workspace; above that, the most recently confirmed are held
speculapro_decision_sanctions_set_truncated_workspaces	Gauge		Workspaces with more confirmed sanctions matches than this process holds for degraded decisions. Above zero, a degraded decision in those workspaces knows only the most recently confirmed; the rest still block when the checks finish. Should be 0
speculapro_decision_sanctions_set_refreshed_timestamp_seconds	Gauge		When this process last re-read the confirmed sanctions matches (Unix time). time() minus this is the age of what a degraded decision relies on; it is re-read at half of SP_DECISION_STATE_TTL (every 5 seconds by default)

Receiving and checking data:

Metric	Type	Labels	What it counts
speculapro_ingest_items_total	Counter	kind (transaction, customer, account, fx_rate, account_event), status (accepted, duplicate, rejected, retry, or error)	Items received through the ingest API and file imports, by the status each was given. error counts once for a whole request that failed, for example because the database was unreachable
speculapro_stream_events_total	Counter		Transactions checked by post-event monitoring (the stream role)
speculapro_stream_batch_duration_seconds	Histogram		Time the stream evaluator took over one batch, from receiving it to having recorded its hits, including any retries

Rules (processes running the stream or decision role):

Metric	Type	Labels	What it counts
speculapro_rule_explanation_errors_total	Counter	rule (the rule key, for example fraud.profile_change_then_payout)	Computed placeholders in a rule’s explanation that failed when the rule fired: a type mistake the save check couldn’t see, a division by zero, or the cost limit. The hit still counts and the alert is still raised; only that part of the explanation reads —

HTTP (processes running the api, ingest or decision role):

Metric	Type	Labels	What it counts
speculapro_http_requests_total	Counter	class (2xx, 3xx, 4xx, 5xx)	HTTP requests answered, by status class. Health probes aren’t counted
speculapro_http_request_duration_seconds	Histogram		Time to answer an HTTP request, for every API route. Health probes aren’t counted
speculapro_login_ratelimit_fallback_total	Counter		Sign-in attempts that were limited by the API process itself because Redis could not answer. While Redis is down each process keeps its own count per address and per account, so the limits still hold (per process rather than across the installation). Any increase means the API cannot reach Redis
speculapro_ratelimit_fallback_total	Counter	limiter (scim, sso, provider)	The same for the other rate limits that guard a way in: SCIM provisioning requests, single sign-on and the provider console’s sign-in. Requests limited by the process itself because Redis could not answer; each process keeps its own count against the same limit until Redis answers again, so a Redis outage narrows these limits to one process instead of lifting them. Any increase means Redis is unavailable to that process

Post-event monitoring and scheduled scenarios:

Metric	Type	Labels	What it counts
speculapro_stream_lag_seconds	Gauge		How far behind post-event monitoring is: the age of the oldest transaction in the batch this stream process fetched last. 0 when there is nothing to do. Each stream process reports its own, so use max
speculapro_scenario_runs_total	Counter	scenario, status (succeeded, failed)	Scheduled scenario runs, in the worker role
speculapro_scenario_failures_total	Counter	scenario, reason (timeout, error, lease_lost, lease_expired)	Scenario runs that failed, and why. Alert on any increase
speculapro_scenario_run_duration_seconds	Histogram	scenario	Time to run one scenario window: reading, hits and alerts
speculapro_scenario_hits_total	Counter	scenario, slot (live, shadow)	Hits written by scheduled scenarios
speculapro_scenario_backoff_versions	Gauge		Scheduled rule versions waiting out a back-off after repeated failed runs (2 minutes, doubling up to 6 hours). Zero in normal operation

Licence (self-hosted installations):

Metric	Type	Labels	What it shows
speculapro_license_state	Gauge	state (valid, expiring, grace, expired, evaluation, unlicensed, invalid, saas)	1 for the installation’s current licence state, 0 for the others. Updated when a process reads the state, at least hourly in the worker role
speculapro_license_expiry_timestamp_seconds	Gauge		When the licence in use expires, as a Unix time; 0 without one

A lapsed licence never affects monitoring; these are for your operations team’s calendar.

Connectors (processes running the connector role; see Connector):

Metric	Type	Labels	What it counts
speculapro_connector_records_total	Counter	topic, result (accepted, duplicate, stale, rejected)	Records read from institution topics. rejected records went to the dead-letter topic
speculapro_connector_lag_records	Gauge	topic	Records waiting on each institution topic for the connector’s group
speculapro_connector_paused	Gauge		1 while reading is paused because the pipeline is behind
speculapro_connector_pipeline_lag_records	Gauge		How far the persister and stream evaluator are behind on SpeculaPro’s transaction topic, as the connector sees it
speculapro_cdc_changes_total	Counter	source_kind (cdc, postgres_cdc), result	Change data capture row changes, by outcome: accepted, duplicate, stale, rejected, skipped, replayed, superseded
speculapro_cdc_slot_lag_bytes	Gauge	source	WAL an institution’s Postgres database keeps for SpeculaPro’s replication slot
speculapro_cdc_slot_behind	Gauge	source	1 while a slot is further behind than SP_CONNECTOR_CDC_SLOT_LAG_ALERT_BYTES
speculapro_cdc_source_failed	Gauge	source	1 while a change data capture source is stopped
speculapro_cdc_last_event_timestamp_seconds	Gauge	source	When the last applied change was committed at the source. time() - this is how stale the source is

Storage and delivery:

Metric	Type	Labels	What it counts
speculapro_persist_rows_total	Counter	table (transactions, decisions, customer_events)	Rows the persister handed to Postgres. For transactions and customer_events this includes rows Kafka delivered again and Postgres ignored as already stored; for decisions only rows newly written are counted
speculapro_notifications_total	Counter	channel (email, sms, whatsapp), result (sent, skipped, failed, retry)	Email, text-message and WhatsApp notifications. skipped means it couldn’t be sent at all: the channel isn’t configured (its SP_NOTIFY_*_PROVIDER is none) or, for a text or WhatsApp message, the person has no phone number for it; retry means the attempt failed and another is scheduled (including each try while a WhatsApp message waits for a logged-out session); failed means the last attempt failed, or WhatsApp said the number isn’t on WhatsApp. WhatsApp verification codes and test messages count here too, as sent or failed (they aren’t retried)
speculapro_notify_whatsapp_session_up	Gauge		Only on worker processes with SP_NOTIFY_WHATSAPP_PROVIDER=gowa: 1 while the GOWA WhatsApp session is connected and logged in, 0 when it is logged out, the device named in SP_NOTIFY_GOWA_DEVICE_ID is missing, or GOWA can’t be reached. Sampled every SP_NOTIFY_GOWA_STATUS_INTERVAL, and set to 0 at once when a send finds the session down or the device missing
speculapro_webhook_deliveries_total	Counter	result (delivered, retry, failed)	Webhook delivery attempts. retry means the attempt failed and another is scheduled; failed means the last attempt failed and the event won’t be retried. See Webhooks

Sanctions and PEP screening (processes running the worker or api role; see Screening lists for operators):

Metric	Type	Labels	What it counts
speculapro_screening_list_fetches_total	Counter	source (the list key, for example un-sc), result (ok, unchanged, older, error, refused)	Fetches of sanctions lists from their publishers. ok: a new version was stored; unchanged: the publisher’s file was the one already held; older: the publisher’s file was older than the version held (for example one loaded from a signed bundle), which stays; error: the fetch failed; refused: the file would have removed too much of the list, so the current version stays until an operator looks. Every fetched list has all five series from start-up
speculapro_screening_list_last_refresh_timestamp_seconds	Gauge	source	Fetched lists only. When the list was last confirmed current, as a Unix time: a successful fetch (changed or not), or a newer version stored from a signed bundle. Only lists that have a version; sampled by the worker at each pass. time() - this is how old the list is. Lists that are imported are not exported here: they are as old as their last import, by design
speculapro_screening_list_entries	Gauge	source	Entries in the current version of each of the installation’s lists that has one, fetched or imported
speculapro_screening_subjects_screened_total	Counter	type (customer, counterparty)	Customers and counterparties screened
speculapro_screening_hits_total	Counter	kind (list, provider)	New potential matches recorded, from the lists or from a provider
speculapro_decision_inline_screenings_total	Counter	result (clear, match, sanctions_match)	Counterparties a decision screened itself because no screening result was stored yet. match is a potential match on a list that is not a sanctions list
speculapro_decision_inline_screening_duration_seconds	Histogram		Time a decision spent screening a counterparty’s name against the in-memory list index
speculapro_decision_inline_records_total	Counter	result (recorded, none, error, dropped)	Potential matches found by inline screening and handed on for review. After error or dropped (the queue was full) the worker records the match when it screens the counterparty
speculapro_screening_provider_requests_total	Counter	provider, result (ok, error)	Calls to screening providers
speculapro_screening_provider_retries_pending	Gauge		Subjects not fully screened: a provider call failed and waits for its retry (after 1 minute, doubling, up to every 6 hours, for as long as the provider stays configured). Every workspace together. See Provider failures and retries
speculapro_model_scores_total	Counter	state (live, shadow)	Transactions scored by a risk model
speculapro_model_trainings_total	Counter	result (succeeded, failed, refused, cancelled)	Risk model trainings. refused is too little data or too many transactions for the chosen window, not a fault
speculapro_model_scheduled_trainings_total	Counter	result (queued, skipped, deferred, error)	Occasions of workspaces’ retraining schedules. skipped: too little data. deferred: another training was in the way, tried again in an hour. error: it could not be queued and stays due
speculapro_api_keys_rotation_due	Gauge		Working API keys whose creator was deactivated and that nobody has rotated or revoked since, across the installation. Sampled by the worker role every minute; no workspace label
speculapro_support_access_requests_total	Counter	result (requested, refused_policy, approved, denied, cancelled, revoked, ended, expired)	Support access requests and what became of them (provider console only)
speculapro_support_access_active	Gauge		Support accesses running now, across the installation. Sampled by the worker role every minute
speculapro_support_session_denied_total	Counter	reason (among them write, scope, route_not_allowed, expired, revoked, ended, workspace_suspended, staff_inactive, handoff_invalid)	Requests a support session, or an attempt to open one, was refused. Worth alerting on write and handoff_invalid
speculapro_provider_logins_total	Counter	result (success, failure, rate_limited)	Sign-in attempts at the provider console
speculapro_provider_actions_total	Counter	action	Entries written to the provider console’s audit log
speculapro_provider_sessions_active	Gauge		Provider console sessions that would still be accepted, sampled every minute
speculapro_provider_workspaces_over_limit	Gauge	limit (users, events)	Active workspaces over what their plan agreed. Nothing is blocked on it
speculapro_provider_workspaces_purged_total	Counter		Workspaces deleted by the worker after their scheduled date
speculapro_provider_purge_waiting	Gauge	reason (disabled, not_suspended, moving, legal_hold, no_dual_control, reports_not_exported, error)	Workspaces whose scheduled deletion date has passed and that are not deleted
speculapro_model_drift_psi	Gauge	state (live, shadow), of (score, feature)	The highest population stability index among the installation’s risk models over the drift window, for the score and for the most shifted feature. Under 0.1 is stable; above 0.25 is a significant shift. Never labelled by workspace
speculapro_models_drifting	Gauge	state	Risk models whose score or a feature has drifted significantly
speculapro_model_load_failures_total	Counter	state (live, shadow)	Attempts to load a risk model in service that failed; the workspace is scored without it until one succeeds
speculapro_models_not_loaded	Gauge	state	Risk models in service that this process could not load when it last tried. A live one means rules guarded by model.available are not firing for that workspace. See Risk models

The source label is always one of the installation’s lists. A workspace’s own imported lists are never a label.

Analytics store (processes running the optional analytics role; see Analytics store):

Metric	Type	Labels	What it counts
speculapro_analytics_lag_seconds	Gauge	source (kafka, postgres)	How far behind the store is. kafka: the age of the oldest record in the batch just written, 0 when caught up. postgres: seconds since the last complete copy of rule hits, alerts and cases
speculapro_analytics_rows_total	Counter	table	Rows written to ClickHouse. Repeated copies are counted, although the store keeps one row
speculapro_analytics_insert_seconds	Histogram	table	How long each batched insert took
speculapro_analytics_errors_total	Counter	stage (insert, rollup, sync, delete, filter, progress, commit, decode, read)	Failures. insert, rollup, sync, delete: a write to the store is failing and being retried. filter, progress, commit: the writer’s check against recorded deletions, its position report, and Kafka offset commits. decode: a Kafka record it couldn’t read and skipped. read: a reader fell back to PostgreSQL
speculapro_analytics_incomplete_workspaces	Gauge		Workspaces whose history hasn’t been copied to the store completely: never copied, or a copy is running or was interrupted. Their long reads use PostgreSQL until speculapro analytics backfill has finished
speculapro_scenario_analytics_reads_total	Counter	scenario	On the worker: scheduled-rule windows read from the analytics store instead of PostgreSQL

Each process also reports the standard Go runtime and process metrics (go_*, process_*): memory, goroutines, CPU time and open files.

Useful queries

Question	PromQL
Decisions per second, by outcome	sum by (outcome) (rate(speculapro_decisions_total[5m]))
Share of decisions degraded	sum(rate(speculapro_decisions_total{degraded="true"}[5m])) / sum(rate(speculapro_decisions_total[5m]))
99th percentile time to decide	histogram_quantile(0.99, sum by (le) (rate(speculapro_decision_duration_seconds_bucket[5m])))
Degraded decisions, by cause	sum by (cause) (rate(speculapro_decision_degraded_total[5m]))
Decided transactions waiting in the spool	max(speculapro_decision_spool_events)
Age of the confirmed sanctions matches a degraded decision relies on, in seconds	time() - min(speculapro_decision_sanctions_set_refreshed_timestamp_seconds > 0)
Workspaces whose confirmed sanctions matches don’t all fit in memory (should be 0)	max(speculapro_decision_sanctions_set_truncated_workspaces)
Items rejected by the ingest API, by kind	sum by (kind) (rate(speculapro_ingest_items_total{status="rejected"}[15m]))
Transactions checked per second by post-event monitoring	sum(rate(speculapro_stream_events_total[5m]))
99th percentile time per stream batch	histogram_quantile(0.99, sum by (le) (rate(speculapro_stream_batch_duration_seconds_bucket[5m])))
Notifications failing, by channel	sum by (channel) (rate(speculapro_notifications_total{result="failed"}[1h]))
Is the WhatsApp session up?	max(speculapro_notify_whatsapp_session_up)
Webhook attempts failing	sum by (result) (rate(speculapro_webhook_deliveries_total[15m]))
Rules whose explanations are failing	sum by (rule) (increase(speculapro_rule_explanation_errors_total[1h])) > 0
Share of HTTP requests failing with 5xx	sum(rate(speculapro_http_requests_total{class="5xx"}[5m])) / sum(rate(speculapro_http_requests_total[5m]))
How far behind post-event monitoring is	max(speculapro_stream_lag_seconds)
Scheduled scenarios failing, by scenario and reason	sum by (scenario, reason) (increase(speculapro_scenario_failures_total[1h]))
Days until the licence expires	(max(speculapro_license_expiry_timestamp_seconds) - time()) / 86400
How old each screening list is, in hours	(time() - max by (source) (speculapro_screening_list_last_refresh_timestamp_seconds)) / 3600
Screening list fetches failing, by list	sum by (source) (increase(speculapro_screening_list_fetches_total{result="error"}[6h]))
Screening list fetches refused because they would remove too much, by list	sum by (source) (increase(speculapro_screening_list_fetches_total{result="refused"}[6h]))
Worst drift of any live risk model	max(speculapro_model_drift_psi{state="live"})
Risk model trainings that failed in the last 6 hours	sum(increase(speculapro_model_trainings_total{result="failed"}[6h]))
Live risk models a process can’t load	max(speculapro_models_not_loaded{state="live"})
Share of calls to each screening provider failing	sum by (provider) (rate(speculapro_screening_provider_requests_total{result="error"}[15m])) / sum by (provider) (rate(speculapro_screening_provider_requests_total[15m]))

You can also ask Redis for the spool’s length directly: LLEN "spool:{decisions}".

Tracing

A trace shows one request’s journey through SpeculaPro as a tree of timed steps, called spans: the HTTP request, each database query, each Redis command, each message handed to Kafka. When a decision is slow or an ingest call fails, the trace shows which step was responsible.

Tracing is off until you set OTEL_EXPORTER_OTLP_ENDPOINT. SpeculaPro then exports traces over OTLP/gRPC, using the standard OpenTelemetry settings; see Configuration: tracing. At high volume, sample: OTEL_TRACES_SAMPLER=parentbased_traceidratio with OTEL_TRACES_SAMPLER_ARG=0.01 keeps one trace in a hundred.

What is traced

Step	Span name	What it records
HTTP requests	The method and route, for example POST /api/v1/decisions or GET /api/v1/customers/{id}	Status, the request path (without the query string), the server, the caller’s address (as SpeculaPro decided it; see below) and user agent. /healthz and /readyz aren’t traced
PostgreSQL	The query	The SQL statement with its placeholders ($1, $2), never the values sent with it; the database host, name and user
Redis / Valkey	The command name, for example evalsha or hmget	The command name and server only, never keys or values
Kafka	The topic, with publish or receive	Topic, partition, offset and consumer group. Never the message key or contents
Batch processing	The topic, with process, for example sp.txn.ingested process	One span for each batch the persister or stream evaluator handles, with the batch size. Its database writes, Redis updates and rule evaluation appear beneath it

Trace context travels in Kafka record headers, so a transaction keeps its trace across Kafka. Its trace shows the HTTP request (an ingest call or a real-time decision), the hand-over to Kafka, and the moment the persister and the stream evaluator each receive it.

The persister and the stream evaluator then work in batches, many transactions at a time. Each batch gets its own trace, whose process span links to the traces of the transactions in it, up to 128 per batch. So from a slow batch you can open the transactions it held, and from a transaction you can follow it to the batch that stored and checked it, in any tracing tool that shows span links (such as Grafana Tempo or Jaeger).

What isn’t linked

A batch of more than 128 transactions links only the first 128. Account events work the same way: each batch the persister stores gets its own process span, linked to the events’ traces.

Request logs carry the trace ID

Each request’s log line ("msg":"http") includes trace_id when the request has one, so you can jump from a log line to its trace:

{"level":"INFO","msg":"http","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736","method":"POST","path":"/api/v1/decisions","status":200,…}

A request that arrives with a traceparent header gets that trace’s ID in the log even when tracing is off.

What reaches your collector

Traces never contain customer names, amounts, transaction details, your account or customer references, SQL values, Redis keys or values, Kafka message keys or contents, or URL query strings (where searches put what people type). They do contain:

• Callers’ IP addresses and user agents, from HTTP spans: your staff’s browsers and your systems. The address is the one SpeculaPro decided on, the same as in the audit log: the connecting machine, or with SP_HTTP_TRUST_PROXY=true the address your own proxy recorded (see SP_HTTP_PROXY_HOPS). Forwarding headers are removed before tracing sees the request.
• Request paths, which contain SpeculaPro’s internal IDs (such as a customer’s or case’s UUID), never your references or names.
• Error messages from PostgreSQL, Redis and Kafka on failed steps. These are normally about the database, not the data, but a database error about a malformed value can quote that value.

Treat the collector like your logs

IP addresses can be personal data, and an error message can occasionally quote a value. Run the collector and trace storage inside your network, protect them like your logs, set a retention period, and include them in your data protection records.

Upgrading from an earlier version

Before the October 2026 fixes, Kafka spans recorded the message key (your workspace ID with the account or customer reference), and HTTP spans took the caller’s address from X-Forwarded-For even when the proxy wasn’t trusted. If you ran tracing on an earlier version, traces already stored may contain those references and client-chosen addresses; apply your retention period to them.

Logs

SpeculaPro writes logs to standard output, one event per line. Collect them with your platform’s usual tooling (journald, Docker or Kubernetes log drivers, Fluent Bit, Vector) and ship them to your log system.

Setting	Values	Production advice
SP_LOG_FORMAT	json (default) or text	json, so your log system can parse fields
SP_LOG_LEVEL	debug, info (default), warn, error	info

Every HTTP request is logged with its method, path, status, duration, client IP and request ID, plus trace_id when the request has a trace (see Tracing):

{"time":"2026-10-09T09:14:02.511Z","level":"INFO","msg":"http","method":"POST","path":"/api/v1/auth/login","status":200,"dur":84211000,"ip":"10.20.0.15","req_id":"web-01/aZ3k…-000042"}

dur is in nanoseconds. Behind a load balancer, set SP_HTTP_TRUST_PROXY=true so ip shows the real client address, but only if the load balancer is yours, and set SP_HTTP_PROXY_HOPS to the number of your proxies that add to X-Forwarded-For. See Configuration: HTTP server.

Messages from the background roles to watch for:

Message	Level	Meaning
persister started	INFO	One line per persist worker at start-up
persist: batch failed, retrying	ERROR	Postgres refused a batch. The persister retries the same batch with growing pauses (up to 30 seconds) and goes no further until it succeeds. See Persister
persist: sent undecodable records to dead-letter topic	WARN	Records that couldn’t be read were set aside in <prefix>txn.ingested.dlq
persist: dead-letter publish failed	ERROR	Those records couldn’t even be set aside. Check Kafka
stream evaluator started	INFO	One line per stream evaluator at start-up
stream: rules loaded	INFO	A workspace’s rule set was (re)loaded, with the number of enabled rules. Appears after every rule change
stream: batch failed, retrying	ERROR	A batch couldn’t be evaluated and is retried with growing pauses (up to 30 seconds); alerts are delayed until it succeeds. See Stream evaluator
stream: rule error	WARN	One rule failed on one transaction and was skipped. The line names the rule
stream: commit offsets	WARN	Kafka didn’t take the acknowledgement; the batch will be delivered again, harmlessly
worker: created partitions	INFO	New monthly partitions were made (normal, about once a month per table)
worker: ensure partitions / worker: expire ingest keys / worker: expire webhook history	ERROR	A maintenance task failed. It will be retried at the next interval
ctr: drafting is on but the workspace has no CTR threshold, so no CTR is drafted; …	ERROR	A workspace has cash transaction report drafting on and no threshold to report against, with tenant, jurisdiction, country and base_currency. Logged for each such workspace at most every six hours. Someone must set the thresholds
ctr: the workspace's CTR thresholds are another currency's figures read in its base currency, so almost no CTR is drafted; …	ERROR	An upgrade kept this workspace’s CTR thresholds from its country’s currency (see Workspaces that never picked a jurisdiction), with tenant, country, base_currency and thresholds_currency. Logged for each such workspace at most every six hours, until a threshold is changed. Someone must correct the thresholds under Settings → Reporting
decision: degraded	WARN	A decision’s checks returned an error (for example Redis was unreachable), and the failure policy answered. The line gives the workspace, the transaction’s external_id and the error. Decisions that simply ran out of time or were turned away by SP_DECISION_MAX_INFLIGHT aren’t logged one by one; watch speculapro_decision_degraded_total for those
decision: refresh confirmed sanctions matches; degraded decisions use the previous sets	ERROR	The background re-read of confirmed sanctions matches failed (logged once per outage). Degraded decisions keep using the last copy, so a match confirmed since then isn’t in it
decision: more confirmed sanctions matches than the in-process set holds; …	ERROR	A workspace has more than 100,000 parties with a confirmed sanctions match. A degraded decision knows only the most recently confirmed 100,000; the rest still block when the checks finish. With tenant and held
decision: Kafka unavailable, event spooled	WARN	Kafka couldn’t take a decided transaction; it was parked in Redis and will be replayed
decision: replayed spooled events	INFO	Parked transactions reached Kafka, with the count
decision: event LOST: Kafka and the Redis spool both failed	ERROR	A decided transaction was lost. The line gives the workspace and external_id, so you can send that transaction again through the ingest API
decision: spooled events LOST while re-queueing	ERROR	Parked transactions were lost while being put back after a failed replay, with the count
decision: spooled event LOST: unreadable	ERROR	A parked entry in the spool couldn’t be read and was dropped, with its size. Counted in speculapro_decision_events_lost_total
decision: events still unsent at shutdown; spooling them	WARN	On a clean stop, Kafka didn’t take every buffered event within 10 seconds; the rest were parked in the spool and will be replayed
decision: shutdown timed out before every event was delivered or spooled	ERROR	On stop, some events were still neither delivered nor parked after the wait. Check speculapro_decision_events_lost_total and Kafka
decision: store for replay and statistics	WARN	Redis didn’t store a batch of answers for retries and the Decisions page statistics. A retry of those decisions is decided again instead of replayed
webhook dispatcher started	INFO	The worker is delivering webhooks
webhooks: endpoint disabled after sustained failures	WARN	An endpoint was switched off. Administrators get a notification too
webhooks: dispatch / webhooks: record attempt	ERROR	The worker couldn’t read or update the delivery queue in Postgres. Deliveries resume when Postgres is reachable
worker: gave up undeliverable webhooks	INFO	Events waiting for an endpoint that has been switched off for the whole SP_WEBHOOKS_RETENTION were marked failed
tracing enabled	INFO	Traces are being sent to the OTLP endpoint. See Tracing
metrics listener failed; stopping	ERROR	The metrics port couldn’t be opened, so the process stopped. See Prometheus metrics
speculapro stopping after a component failed	ERROR	One part of the process failed, and the whole process stopped so it can be restarted cleanly. The line gives the error

A failed dependency check looks like:

{"time":"…","level":"WARN","msg":"dependency unhealthy","dep":"postgres","err":"…"}

Logs contain IP addresses

Request logs include client IP addresses, which may be personal data. Apply your data protection policy to log retention and access, as you would for any other system.

Multi-region

A cell of a multi-region deployment also logs cell: the directory could not be reached; will retry (WARN) while the directory is away. The cell keeps serving; only creating and moving workspaces need the directory. There are no metrics specific to cells or the directory yet. A single installation never logs this.

What to alert on

Alert	Source	Urgency
/healthz failing	Uptime monitor / orchestrator	Page someone: SpeculaPro is down
/api/v1/system/health returns 503 for more than a few minutes	Uptime monitor	Page someone
persist: batch failed, retrying repeating	Logs	Page someone: received data is not reaching Postgres
Persister consumer lag growing (group speculapro-persist on <prefix>txn.ingested)	Kafka / Redpanda monitoring, e.g. Redpanda Console or rpk group describe	Investigate: the persister is falling behind or stopped
stream: batch failed, retrying repeating	Logs	Page someone: transactions are not being checked and alerts are delayed
Stream consumer lag growing (group speculapro-stream on <prefix>txn.ingested)	Kafka / Redpanda monitoring	Investigate: alerts are arriving late
stream: rule error	Logs	Investigate within a day: a rule is broken and is being skipped
Any increase in speculapro_decision_events_lost_total	Metrics	Page someone: decided transactions aren’t being stored or monitored. Check Kafka and Redis, then resend the transactions named in the event LOST log lines
Share of degraded decisions above what you agreed with the business (for example 1% for 5 minutes)	Metrics	Page someone: payments are being decided by the failure policy, not your rules. The cause label says why
99th percentile decision time close to the workspace’s budget	Metrics	Investigate: you are running out of capacity. See The decision role
max(speculapro_decision_spool_events) above zero for more than a few minutes, or growing	Metrics	Investigate: Kafka is down or refusing writes. See When Kafka is down
speculapro_webhook_deliveries_total{result="failed"} increasing	Metrics	Investigate within a day: a receiver lost events after every retry. Check the endpoint’s deliveries under Settings → Webhooks
Any increase in speculapro_rule_explanation_errors_total, especially after a rule version goes live	Metrics	Investigate within a day: the rule label names a rule whose explanation shows — instead of a figure. Alerts are unaffected, but analysts lose part of the reason. Ask the rule author to fix the placeholder in a new version, and to Test it with sample data first
Anything arriving on an institution dead-letter topic (<topic>.dlq), or speculapro_connector_records_total{result="rejected"} increasing	Kafka monitoring / Metrics	Investigate within a day: those records are not monitored. Fix them and publish them again. See Kafka topics
speculapro_connector_paused at 1 for more than 30 minutes	Metrics	Investigate: the persister or stream evaluator can’t keep up. Scale them
max by (source) (speculapro_cdc_slot_behind) at 1 for 5 minutes (Helm: SpeculaProCDCSlotBehind)	Metrics, plus a notification to data-source managers	Page someone: an institution’s database is keeping WAL for SpeculaPro and its disk can fill up. Fix the connector, or drop the slot. See Postgres replication for DBAs
speculapro_cdc_source_failed at 1 for 5 minutes (Helm: SpeculaProCDCSourceFailed)	Metrics, plus a notification to data-source managers	Investigate within a day: a change data capture source stopped. Its reason is on the source
rate(speculapro_cdc_changes_total{result="rejected"}[1h]) above zero	Metrics	Investigate within a day: changes are being rejected. Each one is in the source’s dead-letter topic with its reasons

| speculapro_connector_lag_records growing while not paused | Metrics | Investigate: the connector is stuck (pipeline unavailable or dead letters failing). Check its logs | | max(speculapro_notify_whatsapp_session_up) == 0 for 5 minutes | Metrics (the Helm chart’s SpeculaProWhatsAppSessionDown) | Act within hours: WhatsApp messages wait in the outbox and fail once they are older than SP_NOTIFY_WHATSAPP_MAX_WAIT (default 24 hours). Link the phone again in GOWA, or fix SP_NOTIFY_GOWA_DEVICE_ID. See When the WhatsApp session logs out | | speculapro_notifications_total{channel="whatsapp",result="failed"} increasing | Metrics | Investigate within a day: people missed WhatsApp messages. The worker log says why (notify: undeliverable or notify: giving up) | | A data source showing Last run had a problem | Settings → Data sources, or filedrop: run finished with a problem in the logs | Investigate within a day. See Data sources | | Redis/Valkey memory growing week on week | Redis monitoring | Investigate before maxmemory is reached. See Feature store: memory | | Any message on <prefix>txn.ingested.dlq | Kafka monitoring, or the dead-letter log line | Investigate within a day | | dependency unhealthy warnings | Logs | Investigate; correlate with dependency monitoring | | Rising HTTP 5xx rate | Logs (status ≥ 500) | Investigate | | Rising HTTP 429 on /api/v1/auth/login | Logs | Possible password-guessing attack. See Sign-in and MFA | | PostgreSQL replication lag, disk space, connection count | PostgreSQL monitoring (Pigsty includes Prometheus and Grafana dashboards) | Per your database team’s thresholds | | Backup job failed | Backup tooling | Fix within a day. See Backups | | Kafka disk usage or under-replicated partitions | Kafka / Redpanda monitoring | Per your platform’s thresholds | | Redis / Valkey memory, failovers | Redis monitoring | Investigate | | TLS certificates expiring | Certificate monitoring | Renew well before expiry | | max(speculapro_analytics_lag_seconds) above 600 for 15 minutes (Helm: SpeculaProAnalyticsLag) | Metrics | Investigate: the store is behind, so long backtests and long scheduled rules are reading PostgreSQL instead, and beyond 15 minutes the Analytics page drops to 3 months. Check the analytics role and ClickHouse. Nothing is lost, and nothing on the decision path is affected | | speculapro_analytics_errors_total{stage=~"insert\|rollup\|sync\|delete"} increasing for 15 minutes (Helm: SpeculaProAnalyticsErrors) | Metrics | Investigate within a day: writes to the analytics store (inserts, the volume totals, the copy from PostgreSQL or deletions) are failing and being retried. Check ClickHouse’s disk and credentials | | max(speculapro_stream_lag_seconds) above 60 seconds for 10 minutes | Metrics | Investigate: alerts are appearing late. Scale the stream role (up to the number of topic partitions), or check Redis and PostgreSQL | | Any increase in speculapro_scenario_failures_total | Metrics | Investigate within a day: a scheduled scenario didn’t complete, so its window wasn’t checked. The reason label says why | | speculapro_scenario_backoff_versions above 0 for an hour (Helm: SpeculaProScenarioBackoff) | Metrics | Investigate within a day: those scheduled rules aren’t checking their windows. The rule’s Schedule and runs panel shows the last error and the next try | | time() - max by (source) (speculapro_screening_list_last_refresh_timestamp_seconds) above 48 hours for 30 minutes (Helm: SpeculaProScreeningListStale) | Metrics | Investigate within a day: customers and counterparties are screened against an old copy of that list. Check the worker’s outbound access, or import a newer bundle on an air-gapped installation. The metric covers fetched lists only; on an air-gapped installation a fetched list is refreshed by loading a bundle, so set the threshold to your bundle cadence. See Screening lists for operators | | speculapro_screening_list_fetches_total{result="error"} increasing with no ok, unchanged or older fetch of the same source in 6 hours (Helm: SpeculaProScreeningListFetchFailing) | Metrics | Investigate within a day: the publisher is down or moved the file, or the worker has no route to it. The error is on Settings → Screening | | speculapro_screening_list_fetches_total{result="refused"} increasing over 6 hours (Helm: SpeculaProScreeningListRefused, critical) | Metrics | Investigate the same day: a publisher’s file would remove more than 20% of a list, so that list is no longer refreshed. Check the publisher’s announcement, then accept the version or leave it | | More than 20% of speculapro_screening_provider_requests_total with result="error" for a provider, over 15 minutes (Helm: SpeculaProScreeningProviderErrors) | Metrics | Investigate within a day: that provider’s results are missing; list screening is unaffected. The subjects it could not be asked about wait for a retry (speculapro_screening_provider_retries_pending). Check its credentials and outbound access from api and worker | | speculapro_screening_provider_retries_pending above 0 for several hours | Metrics | Investigate within a day: those customers and counterparties are screened against the lists only until the provider answers. Fix the provider, or disable or remove it if it is no longer used | | max(speculapro_model_drift_psi{state="live"}) above 0.25 for 6 hours (Helm: SpeculaProModelDrift) | Metrics | Investigate within a few days: a live risk model’s traffic no longer looks like its training data, so rules that read its score fire differently. The worker log (models: significant drift) names the workspace; its compliance team trains a new version | | speculapro_model_trainings_total{result="failed"} increasing (Helm: SpeculaProModelTrainingFailing) | Metrics | Investigate within a day: usually the worker’s memory limit against SP_MODELS_MAX_MEMORY_MB, or SP_MODELS_TIMEOUT | | max(speculapro_api_keys_rotation_due) above zero for 24 hours (Helm: SpeculaProAPIKeysRotationDue, warning) | Metrics | Act within days: someone who created API keys was deactivated and the keys still work. Settings → API keys in the workspace shows which; rotate or revoke them, or turn on automatic revocation. See Managing API keys | | speculapro_provider_workspaces_over_limit above zero for 1 hour (Helm: SpeculaProWorkspaceOverPlanLimit, info; only with the provider console on) | Metrics | No urgency: a workspace is over its agreed people or monthly transactions. Nothing is blocked. The console’s workspace list shows which. See Plans and limits | | speculapro_provider_purge_waiting above zero for 24 hours (Helm: SpeculaProWorkspaceDeletionWaiting, warning) | Metrics | Act within days: a scheduled deletion’s date passed and the workspace is not deleted. The reason label and the workspace’s page in the console say why. See Suspending and deleting | | max(speculapro_models_not_loaded{state="live"}) above zero for 2 minutes (Helm: SpeculaProLiveModelNotLoaded, critical) | Metrics | Act the same day: a workspace’s live risk model can’t be loaded, so every rule guarded by model.available has stopped firing there. The log line models: a model could not be loaded names the workspace and the cause. Retire the version or promote another | | speculapro_license_state is 1 for grace, expired, unlicensed or invalid, or the expiry is less than 30 days away | Metrics | Plan the renewal. Monitoring is never affected, but new workspaces, rule versions and pack installs pause after the grace period. See Licensing |

Watch sign-in failures as a security signal

A burst of 429 responses or failed sign-ins from one address, or across many accounts, is worth a look by your security team, even though SpeculaPro already limits and locks repeated attempts.

Ready-made alerts on Kubernetes

The Helm chart can create a ServiceMonitor that scrapes every pod and a PrometheusRule with alerts for degraded decisions, lost decision events, the spool, stream lag, webhook failures, the 5xx rate, decision and HTTP latency, the licence, failing or backing-off scheduled rules, rule explanation errors, change data capture slots falling behind and stopped sources, the WhatsApp session (when GOWA is used), stale sanctions lists, failing list fetches and screening provider errors (when screening is enabled), the analytics store falling behind or failing to write (when ClickHouse is enabled), and unreachable pods. See Kubernetes with Helm: monitoring. On other platforms, use the table above.

Grafana dashboards

SpeculaPro ships four ready-made Grafana dashboards for a Prometheus that scrapes its metrics port. They are JSON files in the Helm chart, under deploy/helm/speculapro/dashboards/.

Dashboard	File	What it answers
SpeculaPro / Overview	speculapro-overview.json	Is it healthy? Traffic, errors, latency and backlog at a glance. Start here
SpeculaPro / Real-time decisions	speculapro-decisions.json	The decision API: answers and time against the 50 ms target, the failure policy, the spool, and sanctions inside the decision
SpeculaPro / Monitoring and background work	speculapro-pipeline.json	Records coming in, the monitoring stream, sanctions screening, risk models, webhooks and notifications going out, and housekeeping
SpeculaPro / Hosted service	speculapro-hosted.json	The provider console and support access. Empty on a self-hosted installation

They follow the four golden signals: traffic, errors, latency and saturation. Each dashboard opens with the figures to look at first, coloured against the same thresholds the chart’s alert rules use by default: decision time at the 99th percentile (50 ms), the share of decisions answered by the failure policy (1%), the share of requests failing with a 5xx (1%), how far monitoring is behind (60 seconds) and model drift (PSI 0.25). The colours don’t follow thresholds you change under prometheusRule.thresholds. The dashboards show the last 6 hours and refresh every 30 seconds.

How to install them with Helm

For Grafana run by kube-prometheus-stack, whose sidecar loads dashboards from ConfigMaps:

1. Set grafanaDashboards.enabled: true in your values.
2. Optionally put them in a folder: grafanaDashboards.annotations: {grafana_folder: SpeculaPro}.
3. Upgrade the release. The chart creates one ConfigMap per dashboard, labelled grafana_dashboard: "1". If your sidecar looks for another label, set it under grafanaDashboards.labels.

serviceMonitor: {enabled: true, labels: {release: kube-prometheus-stack}}
grafanaDashboards:
  enabled: true
  annotations: {grafana_folder: SpeculaPro}

How to import them into any other Grafana

1. In Grafana, open Dashboards → New → Import.
2. Upload a file from deploy/helm/speculapro/dashboards/.
3. Choose your Prometheus data source and select Import. Each dashboard has a Data source variable at the top, so you can switch later.
4. Repeat for the other files.

A panel is empty

A panel shows nothing when Prometheus isn’t scraping the role that exports its metric, or when the feature isn’t in use (no risk model, no screening provider, a self-hosted installation on the Hosted service dashboard). Check the scrape targets first; see Prometheus metrics.

The files are generated: if you change them, edit the panel lists in deploy/grafana/build.py and run python3 deploy/grafana/build.py, or keep your own copy under another name so an upgrade doesn’t overwrite it.

Related

• Troubleshooting
• Production checklist
• Backups and restore
• Scaling and performance