Kubernetes with Helm

In short

The Helm chart (packaged with each release as speculapro-<version>.tgz; deploy/helm/speculapro in the deployment files) runs SpeculaPro on Kubernetes for installations of any size. By default it runs one Deployment per role (api, ingest, decision, stream, persist, worker, connector), so each scales on its own; small clusters can run a single all-in-one Deployment instead.

The chart doesn’t install PostgreSQL, Kafka or Redis. You run those yourself, typically Pigsty for PostgreSQL, a Redpanda or Kafka cluster, and Valkey with Sentinel or Cluster. For a single server, see Docker Compose in production.

The chart’s values are documented line by line in values.yaml, and values.schema.json rejects unknown keys, so a misspelt setting fails the install instead of being silently ignored.

Before you start

You need	Notes
Kubernetes 1.30 or newer	The pods use the built-in preStop sleep action: the image has no shell to run a sleep command
Helm 3
PostgreSQL 18	With the three SpeculaPro roles. See PostgreSQL
Kafka or Redpanda	Three or more brokers for production
Redis or Valkey	Sentinel or Cluster for production. See Redis / Valkey
A verified image digest	See Verifying releases

How to install

1. Prepare PostgreSQL

With Pigsty, declare the database and the three roles in your cluster definition, and apply it with Pigsty’s pgsql-user.yml and pgsql-db.yml playbooks (or at cluster creation):

pg_users:
  - {name: speculapro_owner,  password: '<owner password>',  login: true, comment: SpeculaPro schema owner, runs migrations}
  - {name: speculapro_app,    password: '<app password>',    login: true, pgbouncer: true, comment: SpeculaPro API, row-level security applies}
  - {name: speculapro_system, password: '<system password>', login: true, bypassrls: true, comment: SpeculaPro background roles}
pg_databases:
  - {name: speculapro, owner: speculapro_owner, extensions: [{name: citext}]}

pgbouncer: true on speculapro_app adds that login to PgBouncer, which the app role connects through (port 5433). The owner and system roles stay out of it: they connect directly (port 5436).

Then, as a superuser on the primary, apply the privilege part of deploy/sql/roles.sql (everything after the CREATE ROLE lines, which Pigsty has already done):

sed '/^CREATE ROLE/d' deploy/sql/roles.sql | psql -d speculapro -v ON_ERROR_STOP=1

Without Pigsty, run deploy/sql/roles.sql as a superuser after CREATE DATABASE speculapro, replacing its change-me passwords.

Which port for which login

The background roles (the system role, SP_DB_SYSTEM_URL) and migrations (the owner role) hold session-level advisory locks, so they must reach the primary directly, through Pigsty’s HAProxy service that skips PgBouncer (the default service, port 5436 in Pigsty’s defaults), or through a pool in session mode. The app role (SP_DB_URL, the api and ingest roles) holds no session state and goes through the transaction-mode pool (port 5433): see Direct or through PgBouncer for the two conditions.

SP_DB_URL=postgres://speculapro_app:<password>@pg-speculapro.db.internal:5433/speculapro?sslmode=verify-full
SP_DB_SYSTEM_URL=postgres://speculapro_system:<password>@pg-speculapro.db.internal:5436/speculapro?sslmode=verify-full

2. Create the Secret

The chart never takes secrets in values.yaml. Create one Secret, named speculapro-secrets unless you change secrets.existingSecret, with these keys:

Key	Value
SP_SECURITY_MASTER_KEYS, SP_SECURITY_BLIND_INDEX_KEY	From speculapro keys generate. See Encryption keys
SP_DB_URL	The speculapro_app connection string
SP_DB_SYSTEM_URL	The speculapro_system connection string. The chart gives it to every pod: the background roles work through it, and every role reads the licence through it (see Database roles)
SP_DB_OWNER_URL	The speculapro_owner connection string. Only the migration Job uses it
SP_KAFKA_USERNAME, SP_KAFKA_PASSWORD	Optional, with kafka.saslMechanism
SP_REDIS_USERNAME, SP_REDIS_PASSWORD, SP_REDIS_SENTINEL_PASSWORD	Optional
SP_NOTIFY_SMTP_PASSWORD, SP_NOTIFY_SPALCE_API_KEY	Optional

docker run --rm ghcr.io/spalce/speculapro@sha256:<digest> keys generate
kubectl create namespace speculapro
kubectl -n speculapro create secret generic speculapro-secrets \
  --from-literal=SP_SECURITY_MASTER_KEYS='1:...' \
  --from-literal=SP_SECURITY_BLIND_INDEX_KEY='...' \
  --from-literal=SP_DB_URL='postgres://speculapro_app:...' \
  --from-literal=SP_DB_SYSTEM_URL='postgres://speculapro_system:...' \
  --from-literal=SP_DB_OWNER_URL='postgres://speculapro_owner:...'

Keep a second copy of the encryption keys

Store the two SP_SECURITY_* keys in your secret manager as well. If the Secret is lost and the keys with it, the encrypted customer data in your database can’t be read by anyone, including SpeculaPro.

With External Secrets Operator, set externalSecrets.enabled: true, point externalSecrets.secretStoreRef at your store, and map each key in externalSecrets.data. The chart creates the ExternalSecret before the migration Job runs. helm uninstall leaves it in place: delete it yourself if you remove SpeculaPro.

Any other secret setting can be added under secrets.env, as the variable name and the Secret key it comes from.

3. Write your values

A minimal my-values.yaml:

image:
  digest: sha256:...            # the digest you verified with cosign
kafka:
  brokers: [redpanda-0.redpanda.kafka:9093, redpanda-1.redpanda.kafka:9093, redpanda-2.redpanda.kafka:9093]
redis:
  mode: sentinel
  addrs: [valkey-sentinel.valkey:26379]
  sentinelMaster: mymaster
notify:
  publicUrl: https://speculapro.bank.example
  emailProvider: smtp
  smtp: {host: smtp.bank.example, username: speculapro}
ingress:
  enabled: true
  className: nginx
  host: speculapro.bank.example
  tls: [{secretName: speculapro-tls, hosts: [speculapro.bank.example]}]
http:
  trustProxy: true
  proxyHops: 1                  # 2 if a cloud load balancer sits in front of the ingress controller
bootstrapWorkspace:             # first install only: creates the first workspace
  enabled: true
  slug: acme
  name: Acme Bank
  adminEmail: [email protected]
serviceMonitor: {enabled: true, labels: {release: kube-prometheus-stack}}
prometheusRule: {enabled: true}

Set http.proxyHops to the number of proxies that add to X-Forwarded-For in front of SpeculaPro, or rate limits and the audit log record the wrong client address. See Configuration: HTTP server.

4. Install

helm install speculapro deploy/helm/speculapro -n speculapro -f my-values.yaml

What happens, in order:

1. The migration Job runs speculapro migrate up as the owner role, then applies deploy/sql/grants.sql with psql. If it fails, nothing else is installed. The grants cover the sign-in and API-key lookups, the single sign-on and SCIM lookups, the two functions the data protection purge uses and the function the worker creates monthly partitions with, and they close the installation’s own tables to the app role; they are applied again at every helm upgrade. A chart from before the purge grants were added left the purge failing: see The retention purge on Helm installations.

2. The topics Job runs speculapro topics, creating any missing Kafka topics. It never changes existing ones.

3. The Deployments start.

4. With bootstrapWorkspace.enabled, a Job creates the first workspace. Read the administrator’s temporary password from its log, once:

   kubectl -n speculapro logs job/speculapro-bootstrap

   The password must be changed at first sign-in. Then install the licence: see Licensing.

Layouts

mode	What runs	When
split (the default)	One Deployment per entry under roles: api, ingest, decision, stream, persist, worker, connector, and analytics when you enable it	Production. Each role scales independently
all	One Deployment, sized by all, running every role	Small clusters
directory	Only the global directory of a multi-region deployment: no roles, no Kafka, no Valkey	A hosted service in more than one region. Never for a self-hosted installation

Each role can have its own replicas, resources, autoscaling (a HorizontalPodAutoscaler), pdb (a PodDisruptionBudget), topologySpread, env, extraEnv, extraEnvFrom, extraVolumes and extraVolumeMounts. What isn’t set comes from roleDefaults. A Deployment can run several roles together with runs, for example background: {runs: [stream, persist, worker]}.

The defaults in values.yaml:

Role	Replicas	Notes
api	2
ingest	2
decision	3	SP_REDIS_POOL_SIZE 200; the PodDisruptionBudget keeps at least 2 running
stream	2	SP_STREAM_WORKERS 4 per pod. Workers across all stream pods must not exceed kafka.partitions (64)
persist	2	SP_PERSIST_WORKERS 4 per pod, with the same limit
worker	2	Safe with two: maintenance takes a lock, and webhook and notification batches are leased. It also runs scheduled scenarios, report jobs, backtests and risk model trainings, so its memory limit is 3 GiB. That is exactly what two backtests and one training may use at their limits (SP_BACKTEST_WORKERS, SP_MODELS_WORKERS and their MAX_MEMORY_MB in config): raise it if they can run together
connector	2	Reads institution Kafka topics and SFTP or S3 file drops. Safe with two: file drops are leased and Kafka reading is a consumer group. Files are staged in /tmp while loading, so SP_CONNECTOR_MAX_FILE_BYTES is 900 MiB, below the 1 GiB tmpVolume: raise both together for bigger files, or mount a volume
analytics	1, off by default	Fills the optional analytics store. Enable it with roles.analytics.enabled: true; the chart refuses that without clickhouse.enabled (roles.analytics needs clickhouse.enabled and clickhouse.url). SP_CLICKHOUSE_WORKERS 2 per pod, with the same partition limit as stream. More replicas add Kafka throughput only: hits, alerts and cases are copied by one pod at a time

Pods are spread across zones and then nodes where possible (topologySpread.whenUnsatisfiable: ScheduleAnyway; set DoNotSchedule to insist on it).

Traffic

Service	Selects	For
speculapro	Pods running the api role	The web app and management API
speculapro-ingest	Pods running ingest	The ingest API
speculapro-decision	Pods running decision	Real-time decisions
speculapro-metrics	Every pod	Prometheus scraping

With ingress.enabled, one Ingress serves the web app on ingress.host and, in split mode, routes /api/v1/ingest to the ingest pods.

Real-time decisions are kept apart from people using the web app. Give your payment system its own way in, either:

• decisionService.type: LoadBalancer, a dedicated load balancer (with loadBalancerSourceRanges to allow only your payment systems, and externalTrafficPolicy: Local to keep the caller’s address), or
• ingress.decision.enabled: true, a separate host (ingress.decision.host) that routes exactly POST /api/v1/decisions and /api/v1/feedback to the decision pods.

Send decisions to the decision host, not the web app host

In split mode, the pods behind the main host don’t run the decision role, so a payment system must call the decision Service or the decision host. The decision log the Decisions screen reads (GET /api/v1/decisions) stays on the main host. See Running roles.

Health probes

Every pod has a startup, liveness and readiness probe:

Pods	Probed on	Liveness	Readiness (ready when)
api	The HTTP port (8080)	/healthz: the process is up	/readyz?role=api: PostgreSQL answers
decision	The HTTP port	/healthz	/readyz?role=decision: PostgreSQL answers
ingest	The HTTP port	/healthz	/readyz?role=ingest: PostgreSQL and Kafka answer
mode: all	The HTTP port	/healthz	/readyz?role=all: PostgreSQL answers
persist, connector	The metrics port (9090)	/healthz	/readyz: PostgreSQL and Kafka answer
stream	The metrics port	/healthz	/readyz: PostgreSQL, Kafka and Redis answer
worker	The metrics port	/healthz	/readyz: PostgreSQL answers
analytics (optional)	The metrics port	/healthz	/readyz: ClickHouse, PostgreSQL and Kafka answer

The chart picks each path from the roles a workload runs, so you don’t need to set them. Leave readinessPath empty unless you have a reason to override it.

Why decision, web and mode: all pods don’t check Kafka or Redis. When Kafka or Redis is unavailable, a decision pod keeps answering, with the workspace’s failure policy, and parks decided transactions in the spool. A readiness check on Kafka or Redis would fail on every decision pod at the same moment and take them all out of the load balancer, so your payment system would get connection errors instead of answers, which defeats fail-open. The web app likewise keeps working cases through such an outage. So these pods stay ready while PostgreSQL answers. See Health checks.

Shutdown

When a pod is stopped (a rollout, a node drain), Kubernetes:

1. runs the preStop sleep, preStop.sleepSeconds (10 seconds), while load balancers stop sending it traffic;
2. sends SIGTERM. SpeculaPro then needs up to about 35 seconds: 20 for requests in flight (SP_HTTP_SHUTDOWN_TIMEOUT), 10 to hand buffered decided transactions to Kafka or the spool, and 5 to write the last replay records;
3. kills it after terminationGracePeriodSeconds, 60 by default.

The chart refuses a grace period shorter than the preStop sleep plus 45 seconds:

terminationGracePeriodSeconds (50) must be at least preStop.sleepSeconds + 45 = 55: shutdown takes up to ~35 s after SIGTERM (HTTP drain 20 s + decision event flush 10 s + bookkeeping 5 s)

If you raise SP_HTTP_SHUTDOWN_TIMEOUT, raise terminationGracePeriodSeconds by the same amount. Rolling updates add new pods before removing old ones (maxUnavailable: 0).

Security settings

Setting	Default
User	Non-root, UID and GID 65532
Root filesystem	Read-only. /tmp is a size-limited emptyDir (tmpVolume.sizeLimit, 1 GiB), for uploaded files
Privileges	No privilege escalation, every Linux capability dropped, the RuntimeDefault seccomp profile
Service account token	Not mounted: SpeculaPro doesn’t call the Kubernetes API
Configuration	Non-secret settings in a ConfigMap. Changing it rolls the pods

Network policies

networkPolicy.enabled is on by default:

Traffic	Allowed
Into the HTTP port	From anywhere, or only from networkPolicy.httpFrom (for example your ingress controller’s namespace)
Into the metrics port	Only from the networkPolicy.metricsNamespace namespace (monitoring)
Out, every pod	DNS; PostgreSQL (5432, 6432), Kafka (9092, 9093) and Redis (6379, 26379), narrowed with to: selectors or CIDRs under networkPolicy.egress; anything under networkPolicy.egress.extra
Out, worker pods	Also networkPolicy.egress.webhookPorts (443) to any address, for webhooks and SMS or email providers. Set it to [] to allow none
Out, worker and api pods, for screening	When screening.enabled is true (the default), also networkPolicy.egress.screening (443): worker pods fetch the sanctions lists and call screening providers, api pods call providers for on-demand screening. With an empty to: it reaches the public internet but not private ranges; with screening.airGapped and an empty to: it opens nothing. Setting webhookPorts to [] does not stop list fetches
Out, api and worker pods, with WhatsApp	When notify.whatsappProvider is gowa, also networkPolicy.egress.gowa (port 3000) to your GOWA server. With an empty to: the port is open to any address: narrow it to GOWA’s namespace or pods
Out, api pods, for single sign-on	Also networkPolicy.egress.sso (443) to any address, on by default, so the web app can fetch your identity provider’s discovery document, signing keys (JWKS), token endpoint and SAML metadata. Narrow to: to your identity provider’s addresses when they are fixed, or set its ports to [] if no workspace uses single sign-on
Out, connector pods	Also networkPolicy.egress.connectorPorts (22, 443, 9092, 9093) to any address, for the SFTP servers, object stores and Kafka clusters your administrators connect. Set it to [] to allow none
Out, connector pods, for change data capture	Also networkPolicy.egress.cdc: postgresPorts (5432, 6432) for the PostgreSQL databases it replicates from, and schemaRegistryPorts (8081) for Debezium Avro schema registries. It is separate from egress.postgres, so narrowing that rule to SpeculaPro’s own database doesn’t stop change data capture. With an empty to: the ports are open to any address: narrow it to the institutions’ databases and registries, or set both lists to [] if no workspace uses it
Out, every pod and the migration Job, with the analytics store	When clickhouse.enabled is true, also networkPolicy.egress.clickhouse (9000, 9440). With an empty to: the ports are open to any address: narrow it to where ClickHouse runs
Hook Jobs	DNS, PostgreSQL and Kafka only (and ClickHouse, as above)

Add what your installation needs under networkPolicy.egress.extra, for example SMTP (587), an or an OpenTelemetry collector (4317). Data source connection tests run on the connector pods, so api pods need no access to SFTP servers, buckets or institution brokers.

WhatsApp through GOWA

To send notifications on WhatsApp through your own GOWA server:

notify:
  whatsappProvider: gowa
  defaultCountryCode: "233"
  gowa:
    baseUrl: http://gowa.whatsapp.svc:3000   # required with gowa; a change is audited
    username: speculapro                     # GOWA's APP_BASIC_AUTH user; required
    deviceId: ""                             # only when GOWA holds several devices
    allowPrivate: true                       # GOWA has a cluster address
    timeout: 15s
    statusInterval: 1m
    maxWait: 24h                             # how long messages wait while the session is logged out
networkPolicy:
  egress:
    gowa:
      ports: [3000]
      to: [{namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: whatsapp}}}]

• notify.gowa.username is required: the chart refuses to render without it, because SpeculaPro refuses a GOWA server without basic authentication outside development. Put the password in your Secret (or ExternalSecret) as SP_NOTIFY_GOWA_PASSWORD; SpeculaPro refuses to start with a user name and no password.
• notify.gowa.maxWait (SP_NOTIFY_WHATSAPP_MAX_WAIT, default 24h) is how long WhatsApp messages wait in the outbox while the session is logged out or the device is missing, before they are marked failed.
• The chart offers only gowa or none for notify.whatsappProvider, and spalce or none for notify.smsProvider. The log provider is for development, and SpeculaPro refuses it for SMS and WhatsApp outside SP_ENV=development.
• api pods send verification codes and test messages; worker pods send the notifications and check the session. The gowa egress rule applies to both.
• With prometheusRule.enabled, the chart adds the SpeculaProWhatsAppSessionDown alert.

Sanctions and PEP screening

Screening is on by default and needs no values on a cluster with internet access. The screening values set the SP_SCREENING_* variables on every pod:

screening:
  enabled: true        # false: no screening work, no egress rule, no alerts
  airGapped: false     # true: never fetch a list; signed list bundles and imports only
  allowPrivate: false  # true: a yente server or list mirror on a private address
  fetchTimeout: 5m
  interval: 30s
  batchSize: 1000
  sourceUrls: {}       # list key → your own download address
  maxRemovalPercent: 20     # refuse a list version that removes more than this share…
  maxRemovalMinEntries: 50  # …of a current version with at least this many entries
  #  ke-frc: https://www.frc.go.ke/wp-content/uploads/2026/09/Domestic-List_Kenya.xlsx
networkPolicy:
  egress:
    screening: {ports: [443], to: []}

• screening.sourceUrls is rendered as SP_SCREENING_SOURCE_URLS, in key order. Addresses must be https; the chart refuses plain http unless screening.allowPrivate is true. See When a publisher moves a list.
• screening.allowPrivate: true is refused with deployment: saas: the chart doesn’t render, as SpeculaPro wouldn’t start.
• networkPolicy.egress.screening applies to worker and api pods. An empty to means the public internet without the private ranges, or nothing when screening.airGapped is true. A to you set replaces that default: it is how you reach a provider or mirror on your own network. See Screening lists: with the Helm chart for an example.
• Set these under screening, not as SP_SCREENING_* under config: the chart refuses those, so that the egress rule and alerts always match what the pods run with.
• On an air-gapped cluster, raise prometheusRule.thresholds.screeningListStaleHours (48) to your bundle schedule. The stale-list alert concerns fetched lists only, and with nothing fetched such a list is refreshed only by a bundle that carries a newer version of it.
• The pods refuse to start with interval or fetchTimeout below 1s, batchSize outside 1 to 10000, or a sourceUrls key that is not a fetched list. See What stops start-up.
• The operator’s commands (speculapro screening import-bundle, import-list, source set-url, provider set and the others) need the system role’s database connection (SP_DB_SYSTEM_URL). See Screening lists for operators.
• screening.maxRemovalPercent (20) and screening.maxRemovalMinEntries (50) are rendered as SP_SCREENING_MAX_REMOVAL_PERCENT and SP_SCREENING_MAX_REMOVAL_MIN_ENTRIES: a fetched list or signed bundle that would remove more than that share of a current version of at least that many entries is refused. A refused fetch raises the SpeculaProScreeningListRefused alert. See When a list loses many entries.

Analytics store (ClickHouse)

The analytics store is optional and off by default. The chart never installs ClickHouse: you run it (in the cluster, on other machines, or as a managed service) and give the chart its address. examples/clickhouse-altinity.yaml and examples/clickhouse-bitnami-values.yaml are starting points.

clickhouse:
  enabled: true
  url: clickhouse://clickhouse-speculapro.analytics.svc:9000
  user: speculapro
  existingSecret: speculapro-clickhouse
roles:
  analytics:
    enabled: true
networkPolicy:
  egress:
    clickhouse:
      ports: [9000]
      to:
        - namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: analytics}}

Value	Default	Meaning
clickhouse.enabled	false	Sets the SP_CLICKHOUSE_* settings on every pod, runs ClickHouse’s migrations in the migration Job, opens the network policy and adds the two analytics alerts. With mode: all, the single Deployment then also runs the analytics role
clickhouse.url	empty	Required when enabled. The native-protocol address: clickhouse://host:9000, or port 9440 with TLS. No credentials in it
clickhouse.database	speculapro	The database. The migration creates it
clickhouse.user	speculapro	The user every pod writes and reads with
clickhouse.existingSecret, clickhouse.passwordKey	secrets.existingSecret, SP_CLICKHOUSE_PASSWORD	Where its password comes from
clickhouse.migrationUser, clickhouse.migrationPasswordKey	empty, SP_CLICKHOUSE_MIGRATION_PASSWORD	A separate, more privileged user for the migration Job only (it creates tables and, with readerRole, roles and row policies). Empty: the Job uses user. With readerRole, also run speculapro analytics rebuild as this user: emptying the store needs TRUNCATE, which the reader role doesn’t grant
clickhouse.readerRole	empty	Switches on row policies. The migration creates the role and grants it SELECT, INSERT and ALTER DELETE on the database. After the first migration, once, grant the role to user and make it that user’s default role; no other grant is needed
clickhouse.tls.enabled, caFile, certFile, keyFile, serverName	off	TLS to ClickHouse. The files are paths inside the pod: mount them with extraVolumes and extraVolumeMounts, as for Kafka below
clickhouse.backtestThreshold, clickhouse.scenarioThreshold	720h	Backtests spanning more than this, and scheduled-rule windows at least this long, read ClickHouse when the workspace’s copy is complete and up to date. Write them as whole hours, such as 720h, and at least 1h
roles.analytics.*	off, 1 replica, 500m CPU and 512 MiB requested, 2 GiB limit	The analytics Deployment in mode: split
prometheusRule.thresholds.analyticsLagSeconds	600	How far behind the store may fall before SpeculaProAnalyticsLag fires

Any other SP_CLICKHOUSE_* setting goes under config, or under roles.analytics.env when only the analytics pods need it. After the first install, copy your existing history once with speculapro analytics backfill; see Analytics store: with Helm.

The data protection purge reaches the store

What the retention purge or an erasure removes in PostgreSQL is removed from the copy too, by the worker role. ClickHouse’s own backups are the exception. See how the purge reaches the copy.

Multi-region

The cell and directory values, and mode: directory, are for a hosted service that runs in more than one region. Leave them empty for a single installation: nothing changes and nothing else needs configuring.

Value	Meaning
cell.region, cell.name, cell.publicUrl	This release is one region’s cell (SP_CELL_REGION, SP_CELL_NAME, SP_CELL_PUBLIC_URL)
directory.url, directory.publicUrl, directory.allowPrivate, directory.timeout	The cell’s side of the directory. Its signing key is SP_DIRECTORY_KEY in your Secret
networkPolicy.egress.directory	Lets the cell’s pods call the directory
mode: directory with directory.server.*	This release is the directory itself. With serviceMonitor.enabled, the chart also renders a ServiceMonitor that scrapes the directory’s pods; it renders no alert rules for the directory

The chart refuses mode: directory together with cell.region, the cell values without cell.region, directory.url without cell.publicUrl, and a directory.url that isn’t https. Example files are in examples/: directory-values.yaml, cell-gh-values.yaml and cell-eu-values.yaml. Everything else is in Running more than one region.

Provider console (hosted service only)

Off by default. providerConsole.* switches it on, ports.provider is its own port, a separate Service named <release>-provider serves it, and networkPolicy.providerFrom says who may reach it (empty means nobody). See Setting up the provider console.

Settings that aren’t keys in values.yaml

Put any other SP_* or OTEL_* setting under config (plain values, in the ConfigMap) or, for secrets, under secrets.env. Per role, use roles.<role>.env. For example, to mount a Kafka CA certificate:

kafka:
  tls: {enabled: true, caFile: /etc/kafka-ca/ca.crt}
extraVolumes:
  - {name: kafka-ca, secret: {secretName: kafka-ca}}
extraVolumeMounts:
  - {name: kafka-ca, mountPath: /etc/kafka-ca, readOnly: true}

Monitoring

Value	What it adds
serviceMonitor.enabled	A ServiceMonitor that scrapes /metrics on every pod, with a component label per role
grafanaDashboards.enabled	One ConfigMap per Grafana dashboard, labelled grafana_dashboard: "1" for the kube-prometheus-stack sidecar. Change the label under grafanaDashboards.labels; put the dashboards in a folder with grafanaDashboards.annotations.grafana_folder. Off by default; not rendered in mode: directory
prometheusRule.enabled	A PrometheusRule with the alerts below. Thresholds are under prometheusRule.thresholds; add your own under extraRules

Alert	Fires when	Severity
SpeculaProDecisionsDegraded	More than 1% of decisions answered by the failure policy, for 10 minutes	critical
SpeculaProDecisionEventsLost	Any decided transaction reached neither Kafka nor the spool	critical
SpeculaProDecisionSpoolBacklog	Decided transactions waiting in the spool for 10 minutes	warning
SpeculaProStreamLag	Post-event monitoring more than 60 seconds behind, for 10 minutes	warning
SpeculaProAnalyticsLag	Only with clickhouse.enabled: the analytics store more than prometheusRule.thresholds.analyticsLagSeconds (600 seconds) behind Kafka or PostgreSQL, for 15 minutes. Long backtests and long scheduled rules read PostgreSQL meanwhile, and the Analytics page drops to 3 months once the store is more than 15 minutes behind	warning
SpeculaProAnalyticsErrors	Only with clickhouse.enabled: writes to the analytics store (inserts, the volume totals, the copy from PostgreSQL or deletions) have been failing for 15 minutes. Nothing is lost: the role retries	warning
SpeculaProWebhookFailures	A webhook delivery failed after all its retries	warning
SpeculaProCDCSlotBehind	A Postgres change data capture source’s replication slot has been further behind than SP_CONNECTOR_CDC_SLOT_LAG_ALERT_BYTES (1 GiB) for 5 minutes: the institution’s database is keeping WAL for SpeculaPro and its disk can fill up. Fix the connector, or drop the slot	critical
SpeculaProCDCSourceFailed	A change data capture source has been stopped for 5 minutes (a mapped column dropped, the slot lost). Fix the mapping or resynchronise; nothing after the stop is lost	warning
SpeculaProHTTP5xx	More than 1% of HTTP requests failing with 5xx, for 10 minutes	warning
SpeculaProDecisionLatencyP99	99th percentile decision time above 50 ms (the design target), for 15 minutes	warning
SpeculaProHTTPLatencyP99	99th percentile HTTP time above 1 second, for 15 minutes	warning
SpeculaProLicenseExpiring	The licence expires within 30 days	info
SpeculaProLicenseGraceOrExpired	The licence is in its grace period, expired, not valid, or missing (unlicensed)	warning
SpeculaProScenarioFailures	A scheduled rule’s run failed in the last 30 minutes (timeout, error, or the worker stopped mid-run)	warning
SpeculaProScenarioBackoff	At least one scheduled rule version has kept failing and been backing off for an hour, so it isn’t monitoring	warning
SpeculaProRuleExplanationErrors	A computed placeholder in a rule’s explanation failed at run time in the last hour	info
SpeculaProWhatsAppSessionDown	Only with notify.whatsappProvider: gowa: the GOWA WhatsApp session logged out or unreachable for prometheusRule.thresholds.whatsappSessionDownFor (5 minutes). WhatsApp messages wait in the outbox and fail once they are older than notify.gowa.maxWait (24 hours); see When the WhatsApp session logs out	warning
SpeculaProScreeningListStale	Only with screening.enabled: a sanctions list was last confirmed current more than prometheusRule.thresholds.screeningListStaleHours (48 hours) ago, for 30 minutes. People are screened against an old copy. Fetched lists only: an imported list never fires it. On an air-gapped cluster a fetched list ages from the day its last bundle was loaded	warning
SpeculaProScreeningListFetchFailing	Only with screening.enabled: fetches of a list failed and none succeeded (ok, unchanged or older) in the last 6 hours, for 2 hours. See Screening lists: troubleshooting	warning
SpeculaProScreeningListRefused	Only with screening.enabled: a new version of a sanctions list was refused in the last 6 hours because it would remove more of the list than SP_SCREENING_MAX_REMOVAL_PERCENT allows; fires on the first refusal. See When a list loses many entries	critical
SpeculaProScreeningProviderErrors	Only with screening.enabled: more than prometheusRule.thresholds.screeningProviderErrorRatio (20%) of calls to a screening provider failed over 15 minutes, for 15 minutes. List screening is unaffected; the subjects the provider could not be asked about wait for a retry (speculapro_screening_provider_retries_pending)	warning
SpeculaProModelDrift	A live risk model’s score, or one of its features, has a population stability index above prometheusRule.thresholds.modelDriftPsi (0.25), for 6 hours	warning
SpeculaProModelTrainingFailing	A risk model training failed in the last 6 hours, for a reason other than too little data or a cancellation	warning
SpeculaProLiveModelNotLoaded	A pod has been unable to load a workspace’s live risk model for 2 minutes, so rules guarded by model.available have stopped firing there	critical
SpeculaProTargetDown	A pod’s metrics can’t be scraped, for 5 minutes	warning

See Monitoring for every metric.

Before you change the chart

deploy/helm/validate.sh lints the chart, renders it with the default values and every file in ci/, checks the result with kubeconform, and confirms that a short grace period, a misspelt key and private screening destinations on the hosted service are refused, and that the screening egress rule appears only when it should. Set OFFLINE=1 to skip downloading schemas for custom resources.

Adding roles from future releases

The chart renders every role from the roles map. When a release adds a role, its release notes say what to add: a roles.<name> entry, the role name in values.schema.json and the chart’s role labels, its settings under config and secrets.env, and any network access it needs. The connector and analytics roles were added this way.

Related

• Upgrading
• Running roles
• Verifying releases
• Licensing
• Production checklist