Case studies / AI audit analytics

Simulated case study: fictional company (Northstar Financial Technologies), synthetic data, no real audit or client.

5. AI-assisted audit evidence review and continuous controls monitoring

On 108 held-out cases the rules-plus-mock pipeline found 40 of 43 labelled exceptions (recall 0.93); it sent all 18 ambiguous or contradictory cases to a human. The model is a mock, so this tests the method, not any real AI.

Simulated. Simulated case study: fictional company (Northstar Financial Technologies), synthetic data, no real audit or client. Findings, people, vendors and metrics on this page are invented. Nothing here is a real audit result or an assurance opinion.

NIST AI RMF (conceptual)
Python
JSON Schema
Vanilla JS

Diagram: AI-assisted evidence review architecture — Pipeline in which deterministic rules and a mock model both examine synthetic evidence, output is schema-validated, a human decides, and every step is logged.

Why it matters to the business

The business context. Assurance work is mostly reading evidence: is this access review for the right period, was this change approved before it went live, does this screenshot relate to the control at all? Even the small audit in case study 1 involved 31 evidence items and 71 exception records to trace. Reviewers' attention is the scarce resource, and automation that can handle the repetitive checks is attractive to every audit and compliance leader.

What is at stake. Adopting AI badly is as risky as ignoring it. A tool that suggests conclusions with confidence can be wrong, can be manipulated by the text it reads, and can lead reviewers to stop checking. Sending evidence to an external model also raises privacy questions. The responsible position is neither enthusiasm nor refusal but measurement: what does it get right, where does it fail, and does it know when it is unsure?

Who relies on the work. An audit manager decides whether any of this enters a methodology. Reviewers would use it daily. The Privacy Officer would have to approve any data leaving the company, and the Chief Risk Officer would want evidence, not enthusiasm.

What the business needs from this work.

Deterministic checks kept separate from AI suggestions and from the reviewer's conclusion, so each can be audited.
A human who decides every case, and a log of what was suggested, what was decided and which version of the rules and model produced it.
A fair evaluation on cases the tool was not built on, reported with its limits, instead of a single impressive number.

What this shows about how I work. It builds the pipeline, evaluates it on held-out cases, and reports the uncomfortable parts: the model is a mock, the scores are optimistic, and reviewer acceptance was never measured. Candour about limits is what makes analytics credible in an audit setting.

Approach

Deterministic rules examine the structured record. A second step reads the free-text snippet, extracts fields and proposes a label with a confidence value and a rationale. The model output must pass a strict JSON schema; anything malformed is rejected. Rules and AI results are stored separately and a human reviewer accepts, corrects or rejects each case. Every run writes a hashed audit log with prompt, rules and model versions.

The "model" is a mock. It is a set of regular expressions and keywords, not a language model. No API key is needed and no network call is made. A live adapter exists but is switched off by default, needs three settings and a command-line flag, and was only tested against a stub.

What is automated, and what is not

Automation here means scripts that run the same way every time on synthetic data. It is not a live connection to any real system. The table separates what a script does from what still needs judgement.

Automation in this case study (simulated, synthetic data)
What is automated	How	Where in the repository	What a person still does
Deterministic evidence checks	Rules examine dates, periods, roles, duplicates and malformed values on every case	src/evidence_review/rules.py	Deciding which rules the control needs
Field extraction and classification (advisory)	A mock model, made of keyword and pattern rules, reads each snippet. A live model could be plugged in, but is off by default	src/evidence_review/mock_model.py, adapter.py	Every final decision: accept, correct or reject
Validation and audit trail	Output must pass a JSON schema; each run writes a hashed log of inputs, outputs and versions	src/evidence_review/schema.py, pipeline.py	Reading the log when something looks wrong
Evaluation	Held-out accuracy, with confidence intervals, for three variants on 108 cases	src/evidence_review/evaluate.py	Deciding whether the results justify any real use
Continuous monitoring (simulated)	3 daily rules over a 31-day simulated feed, with exception tracking and simulated alerts	src/ccm/	Triaging alerts; tuning rules to limit false alarms

Re-run it. bash scripts/build_all.sh rebuilds everything from the synthetic data in 22 steps and runs 92 automated tests. The single script is named in each row.

New: the continuous controls monitoring demo shows daily re-performance of three audit tests, with alerts and escalation. It is a replay of a simulated feed, not a live service.

Held-out results

A 44-case development set was used while building the mock. A separate 108-case held-out set (different seed) was run once; labels were set before the run. Distribution: 43 exception, 35 no exception, 18 needs review, 12 not relevant.

Variants on the held-out set
Variant	Precision	Recall	F1	False-positive rate	Abstention (all cases)
Rules only	0.94	0.70 (95% CI 0.55 to 0.81)	0.80	0.057	0.33
Mock model alone	0.95	0.95 (95% CI 0.84 to 0.99)	0.95	0.057	0.08
Rules + mock (assisted)	0.95	0.93 (95% CI 0.81 to 0.98)	0.94	0.057	0.19

The assisted pipeline correctly abstained on all 18 ambiguous or contradictory cases, and rules alone could not tell that 12 snippets were unrelated evidence. The mock model alone scored slightly higher on exceptions (recall 0.95 against 0.93), so adding structured data helped mainly on contradictions. The recall confidence intervals for rules-only and assisted overlap at this sample size.

Open the reviewer queue Read the evaluation report

What a team would do next

Run a real model on real (permitted) evidence under a privacy review, measure reviewer acceptance and correction rates with several reviewers, and only then judge value. Those measurements do not exist here: every case has reviewer decision "Pending: no human review performed", and no efficiency or quality claim is made.

Limitations: why these numbers are optimistic

The corpus and the mock were written by the same person from related templates, so the mock fits the corpus.
Part of the advantage of combining sources is by design: some cases were built so the text reveals what the structured data hides.
Labels are the author's judgement and the irrelevant-evidence cases are easy.
Two post-run fixes were made and logged in the change log: a crash on malformed dates and a metric definition.
Prompt injection, data leakage and over-reliance are treated in the threat table of the README.

Downloads

Every file below is generated by this repository's scripts and listed in the download index.

Evaluation report (PDF)PDF · 18 KB
Evaluation results (CSV)CSV · 49 KB
Evaluation metrics (JSON)JSON · 13 KB
Run manifest (JSON)JSON · 1 KB
Audit log (JSON Lines)JSONL · 118 KB
Output schema (JSON Schema)JSON · 2 KB
Prompt v1.0 (reference only; the mock ignores it)TXT · 2 KB
Pipeline READMEMD · 3 KB
Monitoring READMEMD · 2 KB
Monitoring policy (CSV)CSV · 1 KB
Daily results (CSV)CSV · 3 KB
Exception log (CSV)CSV · 2 KB
Simulated alerts (CSV)CSV · 6 KB
Feed events (CSV)CSV · 2 KB
Summary (JSON)JSON · 1 KB

Sources

S018: Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1 (NIST), verified

See the full source register.

← AI governance · Dashboard