Skip to content

Production checklist

Work through this list before you load real customer data. Each item links to the page that explains it. Print it, tick it, and keep it with your go-live records: auditors like to see it.

Application

Section titled “Application”
  • SP_ENV=production (or another non-development name). This makes encryption keys mandatory, switches off the interactive API docs by default, and sends HSTS headers. → Configuration
  • A released image, deployed by digest: speculapro version prints a real version, not dev. → The image and its commands
  • A verified release, deployed by digest: the image signature checked with cosign verify, and image.digest (Helm) or SPECULAPRO_IMAGE (Compose) set to its sha256: digest. → Verifying releases
  • A licence installed, with its expiry date in the operations calendar and the licence alerts set up. → Licensing
  • SP_LOG_FORMAT=json, with logs shipped to your central log system.
  • SP_ROLES lists only roles that exist: all, or any of api, decision, ingest, persist, stream, worker.
  • SP_METRICS_ADDR bound to an internal interface (or the port firewalled), and every process scraped by your Prometheus. → Monitoring

Encryption keys

Section titled “Encryption keys”
  • Keys generated with speculapro keys generate for this installation (never reused from staging). → Encryption keys
  • Keys stored in a secret manager or a root-only file, not in git or images.
  • Two offline backups of the keys, stored separately from database backups.
  • A named person responsible for the keys, and a written rotation schedule.

Network and HTTPS

Section titled “Network and HTTPS”
  • SpeculaPro is reachable only over HTTPS, through a reverse proxy or load balancer that terminates TLS (nginx, HAProxy, Caddy, Traefik or a cloud load balancer).
  • SP_SECURITY_COOKIE_SECURE=true (the default). The sign-in cookie is then HTTPS-only and named __Host-sp_session.
  • SP_HTTP_TRUST_PROXY=true only if a proxy you control adds to X-Forwarded-For; otherwise false. With it on, SP_HTTP_PROXY_HOPS equals the number of your proxies that add to the header (1 for a load balancer, 2 for a CDN in front of one). → Why it matters
  • SP_HTTP_ADDR binds to an internal address (for example 127.0.0.1:8080), so the proxy is the only way in.
  • Postgres, Kafka and Redis are not reachable from the internet, only from SpeculaPro’s servers and your administrators.
  • SP_HTTP_API_DOCS left unset (off) unless you need it; if on, accept that /api/docs loads its viewer from unpkg.com.

PostgreSQL

Section titled “PostgreSQL”
  • PostgreSQL 18 or later. → PostgreSQL
  • The three roles created, with strong unique passwords.
  • Migrations run as speculapro_owner; the api and ingest roles connect as speculapro_app (SP_DB_URL).
  • Every process has SP_DB_SYSTEM_URL pointing at speculapro_system. Without it, processes running decision, persist, stream, connector, analytics or worker refuse to start, and an api or ingest process can’t read the licence. → Database roles
  • deploy/sql/grants.sql run after the migrations, and again after every upgrade.
  • sslmode=verify-full in SP_DB_URL.
  • High availability (Pigsty/Patroni or a managed service) if downtime matters.
  • Point-in-time backups configured and a restore tested.
  • At least one process runs the worker role, so future partitions are created, old idempotency keys cleared, notifications sent and webhooks delivered. → Worker

Kafka / Redpanda

Section titled “Kafka / Redpanda”
  • At least three brokers and SP_KAFKA_REPLICATION_FACTOR=3 before creating topics. → Kafka
  • Partition count decided (default 64) before creating topics.
  • Automatic topic creation switched off on the brokers.
  • TLS on, and SASL/SCRAM credentials for SpeculaPro.
  • speculapro topics run successfully, and run again after every upgrade (new versions can add topics).
  • Retention on <prefix>txn.ingested longer than any persister outage you could have (a week is a safe start). → Backups
  • The total of SP_PERSIST_WORKERS across all persist processes, and separately the total of SP_STREAM_WORKERS across all stream processes, is no more than the partition count. → Running roles
  • SP_PERSIST_GROUP and SP_STREAM_GROUP are different, and the same on every process of each role. → Consumer groups

Redis / Valkey

Section titled “Redis / Valkey”
  • Sentinel (three Sentinels) or Cluster, not a single node, if real-time decisions matter. → Redis / Valkey
  • A password or ACL user, and TLS when crossing machines.
  • maxmemory set, with monitoring, sized from a load test with your own volumes. → Feature store: memory
  • Persistence (AOF) on, so a restart does not empty the running totals: there is no rebuild command for a running installation. → Redis / Valkey
  • maxmemory-policy noeviction, so Redis never silently deletes running totals, retry answers or spooled decisions. → Redis / Valkey: real-time decisions

Real-time decisions

Section titled “Real-time decisions”

Only if you use the decision API.

  • At least two decision processes behind a load balancer, routing POST /api/v1/decisions and POST /api/v1/feedback to them. → Running roles
  • That load balancer checks decision processes with /readyz or /readyz?role=decision (PostgreSQL only), never a check that includes Kafka or Redis, so an outage of Kafka or Redis doesn’t remove them all at once. → Health checks for decision nodes
  • A shutdown grace period of at least 45 seconds (60 in the Helm chart and Compose file). → What happens on shutdown
  • Each workspace’s decision budget and failure policy (fail-open or fail-closed) agreed with the business and set. → Workspace settings
  • A load test with your own rules and hardware, at your expected peak, with the degraded rate within what you agreed. → Load testing
  • SP_REDIS_POOL_SIZE raised on decision nodes if the load test showed decisions waiting for connections.
  • Alerts on any increase of speculapro_decision_events_lost_total and on the share of degraded decisions. → Monitoring

Webhooks

Section titled “Webhooks”

Only if you send webhooks.

  • SP_WEBHOOKS_ALLOW_HTTP unset or false. SpeculaPro refuses to start with it outside development.
  • SP_WEBHOOKS_ALLOW_PRIVATE left false unless your receivers are on your own network, and set the same on api and worker processes.
  • If it is false, the worker can reach each receiver over the internet (or through your firewall’s egress rules). SpeculaPro doesn’t use an outbound proxy for webhooks.
  • Each receiver verifies signatures, rejects old timestamps and ignores duplicate webhook-ids.

Rules

Section titled “Rules”
  • A rule pack installed, and its rules approved and enabled before go-live. Running totals build up from the first transaction whether or not a rule is enabled, so rules that look back over weeks have history from the start; Redis must be up from day one for that. → Managing rules
  • The lists the rules read (for example high_risk_countries) created and filled. → Lists

Notifications

Section titled “Notifications”
  • SP_NOTIFY_PUBLIC_URL set to the address people use, so links in messages work. → Notifications settings
  • A real email provider: SP_NOTIFY_EMAIL_PROVIDER=smtp (or spalce). The default log only writes messages to the log, so nobody receives them.
  • SP_NOTIFY_FROM_EMAIL on your own domain, allowed by your SPF/DKIM records, and a test message received in a real mailbox (not spam).
  • If people will use text messages: SP_NOTIFY_SMS_PROVIDER=spalce with a registered sender ID. Otherwise leave SMS at none and tell staff that only email works.
  • An alert on the log line notify: giving up. → Worker: notifications

Workspace and people

Section titled “Workspace and people”
  • Workspace created with the right base currency and timezone. Both are hard to change later. → Quickstart
  • The administrator has signed in, changed the temporary password, set up two-factor authentication and saved the recovery codes somewhere safe. → Recovery codes
  • The public https address is set and final, so passkeys are available and stay valid: changing the host name later makes them unusable.
  • Roles assigned with separation of duties: whoever writes rules is not the only approver; analysts cannot close high-risk cases alone. → Roles and permissions

Monitoring

Section titled “Monitoring”
  • Load balancer checks on /readyz?role=<the roles the process serves>, which checks only what those roles need, and liveness checks on /healthz. Both return 204 when healthy; /readyz returns 503 when a needed dependency is down. An uptime check on /api/v1/system/health, which returns 503 when any dependency is degraded. → Health checks
  • persist, stream, worker and connector processes checked on their metrics port (/readyz there checks the dependencies each needs), supervised and restarted automatically, with alerts on persist: batch failed, retrying, stream: batch failed, retrying, stream: rule error, and on the lag of both consumer groups. → Monitoring
  • Alerts on Postgres replication lag, disk space, Kafka consumer lag and Redis memory.
  • On Kubernetes, serviceMonitor and prometheusRule switched on in the Helm chart, with the alerts routed to someone. → Kubernetes with Helm: monitoring

Compliance

Section titled “Compliance”
  • Record retention period confirmed with your regulator (for Ghana, the FIC) and reflected in backups and partition archiving.
  • Data protection registration and hosting location confirmed with your data protection authority.
Section titled “Related”