Quality assurance

In short

A monitoring programme is only as good as the decisions made on its alerts. Quality assurance (QA) means regularly checking those decisions, and the rules behind them, so problems are found by you rather than by an examiner.

This playbook is for MLROs, compliance managers and team leads.

What to check

Area	The question	How
Case decisions	Were closed cases investigated properly and closed with the right disposition?	Sample and re-review closed cases
Documentation	Could an auditor follow the reasoning from the case alone?	Read the closing notes in the sample
Timeliness	Are alerts and cases worked within your SLAs?	Watch the SLA at risk count and review overdue and ageing active cases. See Escalation and SLAs
Reports	Are STRs clear, complete and filed on time?	Review a sample of filed reports against the STR checklist
Rules	Are rules producing useful alerts?	Review dispositions of closed alerts per rule, and the workspace’s 30-day false-positive rate
Suppressions	Are suppressions still justified?	Review active suppressions before they expire: each customer’s page lists theirs (Suppressing alerts), and GET /api/v1/suppressions?state=active lists all of them with their expiry (Suppressions API)
Designations and lists	Are PEP, high-risk and deny-list entries current?	Periodic review of designations (the Flagged tab of Customers) and lists (the Lists panel on Rules)

How to sample closed cases

1. Find closed cases. Open Cases and select the Closed tab. The most recently closed come first, 50 at a time; select Load more at the bottom for the next 50. Search and the priority filter narrow the list (see How to find a case). For a larger sample, export them page by page through the API. Dismissed alerts are on the Closed tab of Alerts, with their outcome. Open one to see who dismissed it, when, and their rationale.
2. Decide the sample each month. A common approach is a fixed percentage of closures per analyst, plus every case in the highest-risk categories (for example PEP, sanctions-related, very large amounts). Set the figures in your AML policy.
3. Over-sample false positives from rules with high volumes. Mass closures are where shortcuts happen.
4. Re-review each case using the general method: would you reach the same conclusion from what’s recorded? Read the outcome and rationale at the top of the case’s Timeline, then the notes and evidence beneath.
5. Grade the result, for example agree, agree but documentation weak, disagree.
6. Act on disagreements: select Reopen on the case if needed (this needs cases.approve; see How to reopen a closed case). The reopening and your reason are recorded on the timeline. Give the analyst feedback.
7. Record the QA itself, with the sample, the grades and the actions, so you can show an examiner your QA programme works.

Review the decision, not the person

QA works best as coaching. Patterns in the findings, such as the same misunderstanding across several analysts, usually point to a gap in training or in a playbook, not to individual failure.

Measuring alert quality

Disposition statistics per rule tell you whether the rule is earning its keep:

Where to read per-rule statistics

The Rule performance panel shows, for each of the ten rules with the most hits, the share of its closed alerts that were not false positives (N% confirmed): on the Command center for recent activity, and on the Analytics page for the period you choose, up to 24 months with the analytics store. For rules outside the top ten, or for a count per disposition, list closed alerts through the API: each one carries its rule_keys and disposition, so a short script or spreadsheet can tally them per rule. Alerts closed with their case show escalated_to_case; the case’s own disposition says how it ended.

Signal	What it suggests
Very high false-positive rate, the same innocent explanation every time	Parameters too broad for some segment. Consider segment-specific parameters, or a time-boxed suppression while you recalibrate
Very few alerts, and none ever reported	Possibly too narrow, or irrelevant to your customer base. Test a looser version in shadow before changing it
A healthy share of true_positive_no_report and report_filed	The rule detects the behaviour it is meant to
Alerts closed very quickly, all as false positives	Possible rubber-stamping. Sample these first
Many alerts closed in one go, with a thin or generic reason	Alerts closed in bulk carry bulk (how many were closed together) in their alert.close audit entry. Sample the large ones, and where the AI assistant is in use, check that the reason says what the person checked and not what the assistant advised

Changes to rules go through the normal version and approval process, ideally tested in shadow first. See Tuning and calibration.

Separation of duties is part of QA

SpeculaPro enforces several checks in the database itself: rule versions, suppressions, high-risk case closures and reports each need a second person. QA confirms the second person is a genuine check and not a formality. Look for approvals given within seconds, or the same pair of people always approving each other.

Annual review

At least once a year, and whenever your business changes significantly (new products, new channels, new markets):

• Review the whole rule set against your institution’s risk assessment.
• Review your SLA targets per priority, and whether high and critical closures (or all closures) need a second approver.
• Review your escalation policies and teams: do they still match who works each level, and how often does a case miss its deadline or get escalated automatically? Each missed deadline is on the case’s timeline and in the audit log as case.sla_breached.
• Run a key-recovery and backup-restore drill if you self-host. See Backups.
• Record the review and its outcome.

Related

• Choosing a disposition
• Tuning and calibration
• Roles and permissions
• Audit trail