File drops (SFTP and S3)

In short

Many banks and payment providers already produce end-of-day files: a statement export, a transaction extract, a customer list. Instead of writing an integration, you can leave those files on an SFTP server or in an S3-compatible bucket (AWS S3, MinIO, Ceph), and SpeculaPro collects them on a schedule.

Each file is loaded once. Every row goes through exactly the same checks, currency conversion, encryption and duplicate protection as a record sent to the ingest API, and gets the same per-row result. When a file is done, SpeculaPro writes a results file next to it (one line per row: loaded, already loaded, or rejected with the reason), and then moves, tags or leaves the original.

Format	Carries	Read as
CSV	Transactions or customers	The same columns as Settings → Data import. See Importing a file: the columns
JSON Lines	Any kind: transactions, customers, accounts, account events, FX rates	One JSON object per line, exactly as the API’s items
ISO 20022 camt.053	Transactions	Bank-to-customer statements, versions .001.02 to .001.13. See ISO 20022 files
ISO 20022 pain.001	Transactions	Credit transfer instructions, versions .001.03 to .001.12. See ISO 20022 files

Any of them may be gzip-compressed (.gz).

File drops are set up per workspace by an administrator, under Settings → Data sources. See Data sources. On a self-hosted installation, the connector role must be running; see Connector.

Push or pull

Everything else in Sending data is push: your systems call SpeculaPro. A file drop is pull: SpeculaPro connects to a server you choose, with credentials you give it. That is why it is off-limits to internal addresses unless your operator allows them, and why an SFTP server’s identity must be pinned. See Security.

How a run works

Every few minutes (you choose: from every 5 minutes to once a day), a connector:

1. Lists the directory (SFTP) or prefix (S3). Only files directly inside are read; sub-folders are not.
2. Skips names it won’t load: hidden files (starting with .), names with /, \, .. or control characters, names over 255 bytes, results files (*.results.csv), and anything that doesn’t match the file name pattern. A pattern such as *.csv also matches name.csv.gz.
3. Leaves for next time any file changed in the last 30 seconds: it may still be uploading.
4. Skips without downloading any file it has already loaded or failed for good, recognised by name, size and its S3 ETag or SFTP modification time.
5. Downloads the file to local disk while computing its SHA-256. The file is never held in memory, and loading reads this stable copy even if the original changes meanwhile.
6. Claims the file by name, size and SHA-256. If the same file was loaded before, or another connector is loading it, it stops here. A file with the same name but different content is a new file and is loaded.
7. Loads it: rows are read as a stream and sent in batches through the ingest checks, on four parallel workers by default. All the rows for one external_id go to the same worker, so a customer’s later version is applied after its earlier one.
8. Writes the results file back (if you set a results path) and keeps a copy for download in the app.
9. Moves, tags or leaves the original (see After loading).

Files are taken oldest first, up to 100 a run by default, so end-of-day files load in the order they were produced.

One file, once

The claim by name, size and checksum is what stops a file being loaded twice, even when several connector processes run at once, and even when you choose to leave files in place. If the same rows arrive again in a different file, the usual duplicate protection applies: each row is recognised by its external_id and reported as already loaded.

A file that keeps stopping a connector is set aside

If loading a file stops without finishing (the connector ran out of memory, was killed or restarted), another connector takes it up again once its claim runs out. After three such stops, the file is marked Failed and skipped, with the message loading this file stopped without finishing 3 times … Check the file, then upload it again under a new name. This is recorded in the file history and the audit log, so one damaged file can’t stop every connector in turn.

How to set up a file drop

You need the Data sources permission (datasources.manage), held by the Administrator role.

1. On the server or bucket, create a directory or prefix for SpeculaPro to read, and give SpeculaPro its own login:
   • SFTP: a user that logs in with an SSH key. SpeculaPro doesn’t use passwords.
   • S3: an access key with permission to list and read the prefix and, depending on what you choose below, to write results, copy and delete objects, or set object tags.
2. Ask the server’s administrator for the SFTP host key: the public key line, such as ssh-ed25519 AAAA…, or its SHA256: fingerprint. SpeculaPro refuses any other server.
3. In SpeculaPro, open Settings → Data sources and select Add data source.
4. Choose SFTP or Object storage (S3), what the files carry, and the format. Leave Detect from the file unless every file has the same format and you want to be strict.
5. Fill in the connection details, the Directory or Prefix, and the File name pattern.
6. Choose a Results path and what happens After loading.
7. Choose how often to look for new files, then select Save.
8. Open the new data source and select Test connection. It connects and counts the matching files, without loading anything.
9. Select Run now to load what is waiting. The File history fills in as each file loads.

Step-by-step screens and every field: Data sources.

File formats

CSV

CSV files use the same columns, headers and date rules as Settings → Data import, so a file that imports by hand also loads as a file drop. CSV covers transactions and customers only. For accounts, account events and FX rates, use JSON Lines.

Each CSV row may be up to 1 MiB, including quoted fields that span lines. A larger row (often a missing closing quote) is skipped while it is read, never held in memory, and reported on its line as row larger than 1048576 bytes (a missing closing quote?); skipped; the rest of the file still loads.

See Importing a file: the columns.

JSON Lines

One JSON object per line, each exactly what you would put in the items array of the matching API call: transactions, customers, accounts, account events or FX rates. Blank lines are ignored, and each line may be up to 1 MiB.

{"external_id":"TX-20261009-000001","occurred_at":"2026-10-09T08:15:00Z","customer_external_id":"C-1001","direction":"out","type":"transfer","channel":"momo","amount_minor":150000,"currency":"GHS"}
{"external_id":"TX-20261009-000002","occurred_at":"2026-10-09T08:16:12Z","customer_external_id":"C-2044","direction":"in","type":"deposit","channel":"agent","amount_minor":50000,"currency":"GHS"}

A JSON array is refused

A file that starts with [ is refused with this is a JSON array; file drops take JSON Lines (one object per line). An array has to be read whole before the first row can load, which doesn’t work for files of millions of rows. Settings → Data import does accept arrays.

ISO 20022

camt.053 statements and pain.001 credit transfer files are read as transactions, with every field mapped as described in ISO 20022 files.

How the format is chosen

With Detect from the file, SpeculaPro looks at the name (after removing .gz), then at the first bytes:

Name ends with	Read as
.csv, .txt	CSV
.jsonl, .ndjson, .json	JSON Lines
.xml	camt.053 or pain.001, from the document’s start; any other XML is refused
Anything else	JSON Lines if it starts with {, ISO 20022 if it starts with <, otherwise CSV

After loading

Choice	What happens to the original	Use when
Move to a processed folder	Moved to the Processed path. If a file of that name is already there, the moved file’s name is prefixed with the time, such as 20261009T180000Z-statement.xml	You control the server, and want the input directory to show only what is waiting
Tag the object (S3 only)	Tagged speculapro-processed=<time>	The bucket is read-only to SpeculaPro apart from tagging, or a lifecycle rule should expire loaded files
Leave it	Untouched	SpeculaPro has read-only access

Moved files can be deleted automatically after a number of days (Keep (days); 0 keeps them). Only files SpeculaPro itself moved there are deleted: anything else in the processed path, such as your own copies, is never touched.

Tagging adds the speculapro-processed tag and keeps the object’s other tags. S3 allows ten tags per object; if an object already has ten, tagging fails and the run reports it, but the file is still recognised as loaded and isn’t loaded again.

Files left in place are never loaded twice

For Leave it and Tag the object, the file stays in the directory, so SpeculaPro keeps a short record of it (name, size and checksum) for as long as it is there, beyond SP_CONNECTOR_HISTORY_RETENTION (90 days by default). After the retention, the record loses its per-line problems and stored results file, but it still stops the file being loaded again. The record is deleted once the file has gone from the directory and the retention has passed.

A file whose rows are still waiting because SpeculaPro’s pipeline was briefly unavailable (status To retry) stays where it is, and the whole file is loaded again on the next run. Rows that were already loaded are recognised as duplicates.

The results file

If you set a Results path, each loaded file gets a results file there named <file name>.results.csv, written uncompressed. On SFTP it is uploaded under a hidden temporary name and then renamed, so your systems never read half a file. It must differ from the input directory; results files are never read as input.

line,external_id,status,errors
2,TX-20261009-000001,accepted,
3,TX-20261009-000002,rejected,expected value to be one of "in, out" (direction: sideways)
4,TX-20261009-000003,duplicate,

Column	Meaning
line	The line in your file (for ISO 20022, the line where the entry or instruction starts)
external_id	The row’s key, if it could be read
status	accepted, duplicate, stale (an older version of a customer or account than one already stored), rejected or retry. See Responses and statuses
errors	Why a row was rejected or must be retried

• Rows are written as they finish, not in file order: sort by line to read them in order.
• A value starting with =, +, -, @, a tab or a carriage return gets a leading ', so a spreadsheet never runs it as a formula. An external_id such as -123 therefore appears as '-123.
• A copy is kept in SpeculaPro for download from the file history, if its compressed size is up to SP_CONNECTOR_MAX_STORED_RESULTS_BYTES (32 MiB by default). It is kept as long as the file history.

Limits

Limit	Default	Set by
Largest file	2 GiB (900 MiB in the Helm chart)	SP_CONNECTOR_MAX_FILE_BYTES. A larger file is recorded as failed once, without being downloaded again
One CSV row	1 MiB	Fixed. A larger row is skipped and reported; the rest of the file loads
Largest content of a .gz file	The same as the largest file	A file that decompresses to more fails with the decompressed file is larger than this deployment’s limit. This stops a small compressed file from filling the disk
Files per run	100	SP_CONNECTOR_MAX_FILES_PER_RUN. The rest load on the next run
One JSON Lines line	1 MiB	Fixed
One ISO 20022 entry or instruction	8 MiB	Fixed. A larger one is skipped and reported; the rest of the file loads
Interrupted loads of one file	3	Fixed. After the third, the file is failed and skipped
Problems listed per file	The first 200, by line	Fixed. The results file lists every row
How often	Every 5 minutes to once a day	The data source’s schedule
Data sources per workspace	50	Fixed

Security

• SFTP servers are pinned. You give the server’s host key when you set it up, and any other key is refused, so nobody can impersonate the server. If the key changes, the run fails with host key mismatch: the server presented SHA256:… (ssh-ed25519), but SHA256:… is pinned: check the new key with the server’s administrator before updating it.
• Keys and secrets are encrypted with your workspace’s key and are never shown again or returned by the API, only whether they are set.
• Internal addresses are refused unless your operator allows them: names such as localhost or *.internal, and private, link-local or cloud-metadata addresses. The check runs when you save and again on every connection, after the name is resolved.
• Plain-http S3 endpoints are refused unless your operator allows them.
• Remote names are never used as local paths. Files are staged under names SpeculaPro chooses.
• Stalled transfers fail after 60 seconds without progress, and connecting gives up after 15.
• Every change, test, run, download and loaded file is audited. See Audit log.

Related

• Data sources
• ISO 20022 files
• Kafka topics
• Importing a file
• Data sources API
• Connector