Troubleshooting

Find the message you see, then follow the fix. SpeculaPro writes errors to standard output; when a command fails, it prints error: followed by the cause.

Start-up

SP_SECURITY_MASTER_KEYS and SP_SECURITY_BLIND_INDEX_KEY are required outside development

SP_ENV is set to something other than development and the encryption keys are missing.

• Production or staging: generate keys with speculapro keys generate and set both variables. Read Encryption keys first.
• Local development: check that SP_ENV is development (or unset; development is the default) in your environment and in .env.

security: blind index key must be 32 bytes, base64

You set SP_SECURITY_MASTER_KEYS but not SP_SECURITY_BLIND_INDEX_KEY, or the value was cut off when copying. The two always go together. Other key errors are listed in Encryption keys.

persist/stream/worker/decision roles and the provider console: database role cannot bypass row-level security

The full message is:

error: persist/stream/worker/decision roles and the provider console: database role cannot bypass row-level security (set SP_DB_SYSTEM_URL to the speculapro_system role)

A process running decision, persist, stream, connector, analytics or worker connects with SP_DB_SYSTEM_URL (or SP_DB_URL if that isn’t set), and that database user can’t bypass row-level security. Point SP_DB_SYSTEM_URL at speculapro_system, which deploy/sql/roles.sql creates with BYPASSRLS. Check with:

SELECT rolname, rolsuper, rolbypassrls FROM pg_roles WHERE rolname = 'speculapro_system';

See Database roles.

unknown role "…"

A typo in --role or SP_ROLES. Valid names: all, api, decision, ingest, connector, stream, persist, worker, and analytics (only with the analytics store). Separate several with commas and no spaces.

config: SP_STREAM_REPLAY_TTL must be at least 1m, got …

SP_STREAM_REPLAY_TTL is set below one minute, for example 30s or a bare 30 (which reads as 30 nanoseconds). Every role refuses to start, not only stream. Set it to at least 1m; the default 30m suits most installations. See Replays and retries.

config: …

A setting has a value of the wrong type, for example SP_HTTP_READ_TIMEOUT=10 (missing the unit; use 10s) or SP_DB_MAX_CONNS=twenty. The message names the variable. See value formats.

config: SP_DECISION_IDEMPOTENCY_TTL must be at least 1h, got …

SP_DECISION_IDEMPOTENCY_TTL is below one hour, or a bare number without a unit. Every role refuses to start. Set it to at least 1h; the default is 72h. See Real-time decisions settings.

config: SP_WEBHOOKS_ALLOW_HTTP is for development only; webhooks must use https

SP_WEBHOOKS_ALLOW_HTTP=true with SP_ENV set to anything other than development. Remove it, and give your receivers an https:// address.

config: SP_WEBHOOKS_TIMEOUT must be between 1s and 1m, got … or config: SP_WEBHOOKS_CONCURRENCY must be between 1 and 256, got …

The setting is outside its allowed range. Use a timeout between 1s and 1m (the default 10s suits most receivers) and a concurrency between 1 and 256 (default 16). See Configuration: webhooks.

listen tcp :8080: bind: address already in use

Another program uses the port. Stop it, or change SP_HTTP_ADDR.

metrics listener :9090: listen tcp :9090: bind: address already in use

Another program, often a second SpeculaPro process on the same machine, already uses the metrics port. The log shows metrics listener failed; stopping and the process exits. Give each process its own SP_METRICS_ADDR (for example :9091), or off. See Prometheus metrics.

Database

db: connect: …connection refused

Postgres is not running or not reachable at the host and port in SP_DB_URL. With Docker Compose, run docker compose ps and wait for the postgres container to report healthy.

…password authentication failed for user …

Wrong user or password in SP_DB_URL. Remember to percent-encode special characters in the password (@ → %40).

…SSL is not enabled on the server or certificate errors

sslmode in SP_DB_URL does not match the server. Locally use sslmode=disable. In production use sslmode=verify-full and point sslrootcert at the right certificate authority.

function uuidv7() does not exist

The server runs PostgreSQL 17 or older. SpeculaPro needs PostgreSQL 18.

permission denied for table …

The tables were created by a different user than speculapro_owner, so the app role’s permissions don’t cover them. See Always run migrations as speculapro_owner.

permission denied for table license_installation

A process is reading the licence as speculapro_app. From migration 00056 the installation’s own tables answer only the system role. Set SP_DB_SYSTEM_URL to the speculapro_system connection for every process, api and ingest included. Until then the licence state is unknown, nothing is blocked, and Settings → License can’t load. See Database roles.

worker: ensure partitions with permission denied

After migration 00056 the worker creates monthly partitions through a function that deploy/sql/grants.sql lets it call. Run grants.sql again as a superuser or the owner. Before 00056, on separate database roles, the same log line said permission denied for schema public; upgrading fixes it.

permission denied for function sp_auth_find_… (nobody can sign in)

deploy/sql/grants.sql has not been run. It must run after the migrations, because the migrations create these functions. See step 5 of How to prepare the database.

The app shows no data, but the data is in the database

The application connects as speculapro_app, so row-level security only shows the signed-in workspace’s rows. This is intended. Check that you are looking at the right workspace.

Kafka

speculapro topics hangs, or kafka: create topics: … with a timeout

SpeculaPro reached the first broker but cannot reach the address the broker advertises. Locally, SP_KAFKA_BROKERS must be localhost:19092 (the external listener), not 9092, which is only reachable inside the Compose network. In production, check that the brokers’ advertised listeners are resolvable from the SpeculaPro servers. See Listeners and addresses.

speculapro topics creates some topics, then fails

The broker ran out of partitions: SpeculaPro creates 16 topics with SP_KAFKA_PARTITIONS each. Lower the number (3 to 6 locally) or give the broker more cores. See How many partitions the broker accepts.

kafka: topic sp.…: …INVALID_REPLICATION_FACTOR…

SP_KAFKA_REPLICATION_FACTOR is larger than the number of brokers. Use -1 or 1 on a single node, 3 on a cluster of three or more.

kafka: unsupported SASL mechanism "…"

SP_KAFKA_SASL_MECHANISM must be empty, PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512.

tls: no certificates in CA file / tls: read ca: …

The file in SP_KAFKA_TLS_CA_FILE (or SP_REDIS_TLS_CA_FILE) doesn’t exist, can’t be read by the SpeculaPro user, or isn’t PEM. A PEM file starts with -----BEGIN CERTIFICATE-----.

tls: client cert: …

Only one of …_TLS_CERT_FILE and …_TLS_KEY_FILE is set, or they don’t belong together. Set both, to a matching pair.

Ingest and persister

Transactions are accepted but never appear in the database

The ingest API answers once a transaction is safely on Kafka; the persister writes it to PostgreSQL afterwards. Check, in order:

1. Is a persist role running? Look for persister started in the logs of some process. SP_ROLES=api,ingest alone stores nothing.
2. Is it failing? Repeated persist: batch failed, retrying lines mean PostgreSQL is refusing the batch: it’s unreachable, out of disk, or a partition or grant is missing. The err field says why. See When a batch keeps failing.
3. Is it just behind? Check the consumer lag of the speculapro-persist group. It should fall back towards zero after a burst. See Consumer groups.

persist: sent undecodable records to dead-letter topic

Something other than SpeculaPro’s ingest API wrote to sp.txn.ingested, or persisters of different versions are running side by side. See Dead letters.

Every ingest request returns 404

The process you are calling doesn’t run the ingest role. Add it to SP_ROLES (or use all), or route /api/v1/ingest/ to the processes that do. See Running roles.

Items come back rejected, retry or stale

These are per-item answers from the ingest API, not server faults. Each one carries the reason. See Responses and statuses.

Old transactions resent months later are stored twice

Duplicate protection lasts SP_WORKER_INGEST_KEY_RETENTION (90 days by default). After that, a resent external_id counts as new. See Worker: what key clean-up means for you.

Rules and alerts

Transactions arrive but no alerts appear

Check, in order:

1. Is a stream evaluator running? One process must run the stream role (all includes it). Look for stream evaluator started in the logs.
2. Are rules enabled? Installing a pack is not enough; each rule must be enabled. A workspace with no enabled rules keeps its running totals up to date but records no hits. See Managing rules.
3. Is the stream evaluator keeping up? Check the lag of the speculapro-stream consumer group. See Stream evaluator: monitoring.
4. Are hits being recorded but not alerted? Hits from shadow versions, suppressed hits, and hits from rules whose action is tag, score_only or allow never create alerts. See Stream evaluator: alerts.
5. Do the rules need a list? A rule that reads a list doesn’t fire until that list exists with at least one item; until then the list reads as empty. See Lists.
6. Is a rule not running at all? Search the logs for rules: approved version no longer compiles (next section).

rules: approved version no longer compiles; rule is NOT running

Logged at ERROR with the workspace (tenant), the rule key (rule), the version ID and the reason (err). Every approved version compiled when it was saved, so this means an upgrade of SpeculaPro changed the rule language in a way that broke this version. The stream evaluator skips that one rule so the rest of the workspace keeps being monitored, and repeats the message each time it reloads rules (about every 15 seconds).

The rule is not monitoring anything until you fix it:

1. Note the rule key and the err text.
2. Create a new version of the rule with the expression corrected, using the CEL reference.
3. Have it approved and deploy it to the active slot. See Managing rules.

Treat this as urgent and alert on it in your log monitoring: a gap in a rule’s coverage is a gap in your monitoring programme. See Operations: monitoring.

stream: rule error

One rule failed on one transaction and was skipped; everything else carried on. The message names the rule. The usual cause is a custom rule reading a custom attribute that the transaction doesn’t carry. (A missing list is not an error; it reads as empty.) Fix the rule (or disable it). See Stream evaluator: a rule that errors.

stream: batch failed, retrying

The stream evaluator can’t finish a batch and keeps retrying it; alerts are delayed until the cause (shown in err) is fixed. See When a batch keeps failing.

Notifications

Notifications don’t arrive

Work through these in order:

1. Is a worker running? Messages are sent by the worker role. With no worker process, they wait in the outbox.
2. Which provider is configured? At start-up the worker logs notify: dispatcher started email=… sms=…. If it says email=log, messages are only written to the log as notify: (log channel); nobody receives them. If it says none, they aren’t sent at all. Set a real provider; see Notifications settings.
3. Is the person set up to receive it? Under Settings → Notifications they must have the channel ticked, and a phone number for text messages. Disabled users receive nothing, and nobody is told about their own actions. See Notifications.
4. Did sending fail? Look for notify: giving up with channel and err. Common causes:
   • smtp: dial …: the mail server can’t be reached (host, port or firewall).
   • smtp: starttls: … or certificate errors: wrong SP_NOTIFY_SMTP_SECURITY for that port, or a certificate the server name doesn’t match.
   • smtp: auth: …: wrong username or password, or the connection isn’t encrypted.
   • smtp: rcpt: …: the mail server refused the recipient or your sender address.
   • spalce: … returned 4xx: wrong API key, merchant code or an unregistered SMS sender ID.
5. Did it arrive but land in spam? Check that SP_NOTIFY_FROM_EMAIL is on your own domain and allowed by its SPF and DKIM records.
6. Do the links point at localhost? You are running with SP_ENV=development or the log provider; set SP_NOTIFY_PUBLIC_URL to the real address. In production SpeculaPro won’t start without it (see below).

A failed message is retried with increasing waits (30 seconds, 1 minute, 2 minutes and so on, up to an hour between tries) until SP_NOTIFY_MAX_ATTEMPTS is reached. To see the state of recent messages, a database administrator can query the notifications table: status is pending, sent, failed or skipped, and last_error says why.

config: SP_NOTIFY_… must be … or … is required …

A notification setting is wrong or missing, so SpeculaPro refuses to start. The message names the setting. The two you are most likely to meet:

• config: SP_NOTIFY_SMTP_SECURITY must be starttls, tls or none: the value has a typo or the wrong case. Use starttls for port 587, tls for port 465.
• config: set SP_NOTIFY_PUBLIC_URL to the address people use to open SpeculaPro; links in notifications point there: a real email or SMS provider is configured outside development, but the public address is empty or still localhost. Set it to the HTTPS address your staff use.

See Notifications settings.

Redis

/readyz returns 503 and logs mention redis (stream processes)

Check SP_REDIS_ADDRS and the password. In Sentinel mode, list the Sentinel addresses (port 26379) and set SP_REDIS_SENTINEL_MASTER. Listing several data nodes without it selects Cluster mode by mistake. See Choosing a mode.

Real-time decisions

Many decisions come back with degraded: true

The workspace’s failure policy is answering because the checks didn’t finish. Look at speculapro_decision_degraded_total by cause:

cause	Means	What to do
timeout	The checks took longer than the workspace’s decision budget	Check Redis/Valkey latency and CPU, and the network between decision nodes and Redis. Raise SP_REDIS_POOL_SIZE on decision nodes, add Redis Cluster shards, or agree a longer budget. See The decision role
dependency	Something failed, usually Redis or, for an uncached customer, Postgres	The log line decision: degraded gives the error. Check /readyz and the dependency
overload	The process already had SP_DECISION_MAX_INFLIGHT decisions in progress	Add decision nodes. Raising the limit only helps if Redis has spare capacity

Decisions are answered but don’t appear on the Decisions page

Decided transactions reach the Decisions page through Kafka and the persister. Check, in order:

1. Is Kafka up? If not, decided transactions are waiting in the Redis spool and will appear once Kafka is back. Check LLEN "spool:{decisions}" in Redis. See When Kafka is down.
2. Is the persister running and keeping up? Check the lag of speculapro-persist.
3. Any decision: event LOST log lines? Those transactions reached neither Kafka nor the spool. Send them again through the ingest API using the external_id in the log line.

Webhooks

Every delivery fails with destination resolves to a private or reserved address

The receiver’s hostname resolves to a private address (10.x, 172.16–31.x, 192.168.x, 127.x and so on). Webhooks are checked at every connection, after DNS, so this happens even if the address looked public when the endpoint was registered. Give the receiver a public address, or, on a self-hosted installation with receivers on your own network, set SP_WEBHOOKS_ALLOW_PRIVATE=true on the api and worker processes. See Which addresses are refused.

Deliveries fail with connection failed or no response within 10s

The worker can’t reach the receiver, or the receiver took longer than SP_WEBHOOKS_TIMEOUT. Check that the worker’s network allows outbound HTTPS to the receiver (SpeculaPro doesn’t use an outbound proxy for webhooks), and that the receiver answers before doing its work. See Answer quickly.

Deliveries fail with HTTP 301 or HTTP 302 (redirects are not followed; use the final URL)

The receiver redirected. SpeculaPro never follows redirects, because a redirect could point anywhere. Change the endpoint’s URL to the final address, including the exact path and a trailing slash if the receiver needs one.

An endpoint shows Disabled on its own

It had failed for SP_WEBHOOKS_DISABLE_AFTER (3 days) without a single success. Fix the receiver, then select Re-enable. See When an endpoint is switched off.

Web app and sign-in

The browser shows UI not built: run make web

The program was built without the web app. This can’t happen with a released image, which always has the web app built in: check that you are running the image you verified, by its digest. (People building from source: run make web, then build again.)

Signing in works but you are immediately signed out

Usually a cookie problem:

• You are on plain http:// with a hostname other than localhost while SP_SECURITY_COOKIE_SECURE=true. Browsers drop secure cookies there. Use HTTPS (recommended), or set SP_SECURITY_COOKIE_SECURE=false for a test system only.
• A proxy is rewriting the Host header or the path. The __Host- cookie requires the exact host and path /.

cross-site request rejected

A browser sent a state-changing request (sign-in, save) from a different origin than SpeculaPro itself. This protects against cross-site request forgery. Open the app on its own address. If a proxy is involved, make sure it passes the original Host header through.

a signed-in request that changes something must carry an Origin or Sec-Fetch-Site header

A request used a session cookie to change something (POST, PUT, PATCH or DELETE) and said nothing about where it came from, so SpeculaPro could not tell it from a forged one and refused it with 403. Browsers always add one of these headers, so this only happens to scripts that sign in and keep the cookie, such as the curl examples for creating an API key. Add the address you reach SpeculaPro on as the Origin header:

curl -b cookies.txt -H "Origin: https://speculapro.example.com" ...

Requests made with an API key (Authorization: Bearer spk_…) are not affected. See Requests that change something.

This temporary password has expired (API: 401 temporary_password_expired)

The person’s temporary password, from an invitation or a reset, is more than 72 hours old. An administrator issues a new one in Settings › People (how). If the person is the only administrator, use speculapro tenant reset-password.

too many attempts, try again later

Too many failed sign-ins. Wait and try again. If everyone is locked out at once behind a proxy, see SP_HTTP_TRUST_PROXY and SP_HTTP_PROXY_HOPS: with too few hops, every request appears to come from your own CDN or load balancer.

Something went wrong on our side. Reference: …

The server hit an internal error. For security, the response doesn’t say what went wrong; the details are in the log. Search the logs of your api processes for the reference after Reference:: it appears as req_id on a line http: internal error, together with the path and the underlying error. See API errors.

/api/docs returns 404

The interactive docs are off outside development. Set SP_HTTP_API_DOCS=true to switch them on.

/api/docs is blank on an isolated network

The docs viewer loads from unpkg.com. Use /api/openapi.json with a local OpenAPI tool instead.

Licensing

License expired: new rules and rule versions are paused … (or installing and upgrading rule packs is paused, or new workspaces are paused)

The licence has lapsed: it is past its grace period, the evaluation ended without one, or the one installed isn’t valid. Monitoring, decisions, alerts and cases are unaffected. Install a valid licence: see Licensing: how to install a licence. speculapro license status shows the state and the reason.

license is not valid: issued for installation …, this is …

The licence is bound to another installation, for example a test one, or the database was replaced by one that isn’t a restore of this installation’s. Send SpeculaPro the output of speculapro license fingerprint from this installation.

license: cannot read SP_LICENSE_FILE

A warning: the file named in SP_LICENSE_FILE doesn’t exist or can’t be read by the container’s user (65532). With Helm, check that the Secret in license.existingSecret exists and that license.key matches the key inside it. Licences installed from the web app still count.

config: SP_DEPLOYMENT must be selfhost or saas or config: SP_LICENSE_CHECKIN_URL must be https

A setting has a value SpeculaPro doesn’t accept. See Configuration: deployment and licensing.

Analytics store (ClickHouse)

The analytics store is optional. None of these stops monitoring, decisions or casework: readers fall back to PostgreSQL.

the analytics role needs SP_CLICKHOUSE_URL

A process was asked to run analytics (in --role or SP_ROLES) without SP_CLICKHOUSE_URL. Set the address, or take analytics out of the list. With --role=all you never name it: it runs whenever the address is set.

config: SP_CLICKHOUSE_URL must be a native-protocol address such as clickhouse://clickhouse:9000

The address must start with clickhouse:// (or tcp://) and name a host. SpeculaPro uses ClickHouse’s native protocol, port 9000 or 9440 with TLS, not the HTTP port 8123. The related config: put ClickHouse credentials in SP_CLICKHOUSE_USER and SP_CLICKHOUSE_PASSWORD, not in the URL means the address contains user:password@. See Configuration: analytics store.

SP_CLICKHOUSE_URL is not set: this installation has no analytics store (everything reads Postgres)

A speculapro analytics … command was run without the setting. With Docker Compose, check SP_CLICKHOUSE_URL is in .env, which the cli service reads.

clickhouse migration 0001_init.sql: … during migrate up

ClickHouse refused a schema statement; the rest of the message is ClickHouse’s own. The usual causes are a user that can’t create databases or tables, and a wrong password. PostgreSQL’s migrations have already been applied by then; fix the cause and run migrate up again. With a clickhouse row policies: … message, the user can’t create roles or row policies: enable access management for it in ClickHouse, or give the migration its own user (Helm: clickhouse.migrationUser).

The Analytics page says Read from the main database, so limited to the last 3 months

The rest of that line says why. Each reason has its own remedy:

The line goes on	Cause	What to do
Longer periods need the analytics store (ClickHouse), which your administrator can connect…	The api process has no SP_CLICKHOUSE_URL	Set it on every process, not only the analytics role
…this workspace’s history hasn’t been copied to it yet…	The workspace had history when the store was switched on (or the store was emptied, or you have just upgraded), and no copy has finished	Run speculapro analytics backfill
This workspace’s history is being copied to the analytics store…	A backfill or a rebuild is running, or one was interrupted	Wait for it. If none is running, run speculapro analytics backfill again
The analytics store holds only this workspace’s most recent history…	The copy is complete from a date 3 months back or less (a backfill run with --from)	Run the backfill without --from
The analytics store is behind at the moment…	The analytics role is more than 15 minutes behind, or stopped	Check the role and speculapro_analytics_lag_seconds
The analytics store can’t be reached at the moment…	ClickHouse couldn’t answer that request	The api log has analytics: ClickHouse read failed; answering from Postgres with the reason: unreachable, wrong credentials, a missing database (run migrate up), or a query that ran past SP_CLICKHOUSE_QUERY_TIMEOUT. On Kubernetes, also check networkPolicy.egress.clickhouse allows the api pods to reach it

speculapro analytics status shows each workspace’s copy and the writer’s position. See When a reader uses the store.

The Analytics page reads the analytics store but greys out 12 or 24 months

The line under the title reads Read from the analytics store, which holds this workspace’s history for the last N months only: longer periods aren’t available. The workspace’s copy is complete from a date, not from the beginning, usually because the backfill was run with --from. speculapro analytics status shows complete from YYYY-MM-DD. Copy the rest with speculapro analytics backfill --workspace <slug>, without --from. See Backfill.

analytics: this workspace's history was never copied to the store; …

The analytics role logs this once per workspace that already had transactions when the store was switched on. Nothing is wrong, but the workspace’s long backtests, long I2 / I3 runs and Analytics page read PostgreSQL until speculapro analytics backfill has run. The gauge speculapro_analytics_incomplete_workspaces counts these workspaces.

analytics: another copy from Postgres is running; try again when it has finished

One backfill, or rebuild from PostgreSQL, runs at a time across the installation. Wait for the other one and run yours again. If the other one was killed, running the command again is all it takes: an interrupted copy is completed by the next one.

analytics: batch failed, retrying

The analytics role couldn’t insert into ClickHouse or refresh the quarter-hour totals, and is retrying, waiting up to 30 seconds between attempts. Nothing is committed to Kafka meanwhile, so nothing is lost. The message carries ClickHouse’s error. Common causes: ClickHouse is down or out of disk; or too many parts (ClickHouse is merging more slowly than it is being written to: raise SP_CLICKHOUSE_BATCH_MAX rather than lowering it). With row policies on, also check the two GRANT and ALTER USER statements were run and migrate up has been run on this release, which grants the reader role INSERT and ALTER DELETE as well as SELECT.

analytics: deletion not applied; it will be retried

The analytics role couldn’t apply a deletion PostgreSQL recorded (an offboarded workspace, or a declined duplicate) to ClickHouse. The record stays, with the number of attempts and the last error, and is tried again on every pass, so nothing is lost. The message carries ClickHouse’s error; the usual cause is ClickHouse being down. speculapro analytics status shows how many deletions are waiting and how many are failing.

analytics: sync pass failed

The copy of rule hits, alerts and cases from PostgreSQL failed for at least one workspace; the line before it (analytics: sync failed) names the table and workspace. The other workspaces were copied, and the next pass tries again from where the last good one ended.

reset consumer group … (stop the analytics role first)

speculapro analytics rebuild --from kafka can’t reset a consumer group that still has members. Stop every process running the analytics role, run it again, then start them. When speculapro analytics status shows the writer caught up, run speculapro analytics backfill: until it has finished, no workspace’s copy is complete and every reader uses PostgreSQL. See How to rebuild the store.

(emptying the store needs TRUNCATE, which the reader role doesn't grant: …)

speculapro analytics rebuild was run as the everyday ClickHouse user on an installation with row policies. Run it with SP_CLICKHOUSE_USER and SP_CLICKHOUSE_PASSWORD of the migration user.

workspace "acme": … from speculapro analytics purge

The workspace no longer exists in PostgreSQL, usually because it has already been offboarded, so its slug can’t be looked up. The message points to --tenant-id: purge by the workspace’s ID instead. No rebuild is needed.

speculapro analytics status      # lists the ID under "not in Postgres any more"
speculapro analytics purge --tenant-id <id> --confirm <id>

Offboarding now removes the copy by itself, so this only concerns a workspace offboarded on an earlier release. See Offboarding: the analytics store.

A long backtest says it read PostgreSQL

Under How this was computed, a run that spans more than SP_CLICKHOUSE_BACKTEST_THRESHOLD says transactions read from the main database and carries a note giving the reason. The results are the same from either database; what you lose is the relief for your PostgreSQL primary.

The note says	What to do
…this workspace’s history hasn’t been copied to it yet (speculapro analytics backfill).	Run speculapro analytics backfill
…a copy of this workspace’s history into it is still running.	Wait for the copy. If none is running, one was interrupted: run the backfill again
…it holds this workspace’s history from a later date than the run needs.	The copy is complete from a date after the run’s warm-up starts. Run the backfill without --from, or from an earlier day
…the analytics role is behind: it hasn’t written every transaction ingested so far.	Check the lag alert and the analytics role. Run the backtest again once it has caught up
Transactions were read from PostgreSQL: the analytics store could not be reached.	Check ClickHouse and the worker’s log (backtest: analytics store unavailable; reading Postgres)

Helm and Docker Compose

terminationGracePeriodSeconds (…) must be at least preStop.sleepSeconds + 45

The chart refuses a grace period too short for a clean shutdown. Leave it at 60, or raise it. See Kubernetes with Helm: shutdown.

values don't meet the specifications of the schema(s)

A key in your values file is misspelt or doesn’t exist; the message names it. The chart rejects unknown keys on purpose, so a typo can’t be silently ignored.

roles.analytics needs clickhouse.enabled and clickhouse.url

roles.analytics.enabled is true but the chart has no ClickHouse to point it at. Set clickhouse.enabled: true and clickhouse.url, or switch the role off. The related clickhouse.url is required when clickhouse.enabled means the address is missing. See Kubernetes with Helm: analytics store.

set secrets.existingSecret (a Secret you created) or enable externalSecrets

The chart needs the Secret holding the encryption keys and database connections. See Kubernetes with Helm: create the Secret.

redis.mode=sentinel needs redis.sentinelMaster or redis.mode=standalone takes exactly one address

redis.mode doesn’t match redis.addrs. Several addresses without a sentinel master mean redis.mode: cluster.

Every decision pod became unready during a Kafka or Redis outage

Their readiness probe checks more than PostgreSQL. The chart’s default for decision pods and for mode: all is /readyz?role=decision or /readyz?role=all, and a bare /readyz on any process that answers decisions also checks PostgreSQL only. So a custom readinessPath naming another role, such as ?role=ingest, is the likely cause: remove the override. See Kubernetes with Helm: health probes.

Single sign-on can’t reach the identity provider on Kubernetes

The chart’s network policies give api pods port 443 to any address for single sign-on (networkPolicy.egress.sso), on by default. If you narrowed networkPolicy.egress.sso.to to your identity provider’s addresses, check they are still right; if you set its ports to [], set them back to [443].

Compose: run ./first-run.sh or set SP_SECURITY_MASTER_KEYS in .env

.env is missing values the Compose file requires. Run ./first-run.sh, which fills in every password and key that is empty. See Docker Compose in production.

Still stuck?

Collect: the output of speculapro version, the exact error, your settings with secrets removed, and the logs around the error at SP_LOG_LEVEL=debug. Then email them to [email protected].

Related

• The image and its commands
• Configuration reference