# AI-assisted audit evidence review (Project 5, SIMULATED)

A prototype that reviews synthetic audit-evidence snippets. Deterministic rules run first; an optional advisory AI step extracts fields and suggests a label. **It runs with no API key and no network**: the default AI step is a deterministic mock, not a language model.

## Setup and run (on the host that has Docker)
```bash
cd <path-to-your-clone-of-this-repository>
docker build -t assurance-build .                       # pinned dependencies in requirements.txt
RUN='docker run --rm --user $(id -u):$(id -g) -e HOME=/tmp -e PYTHONPATH=/work/src:/work/scripts -v "$PWD":/work -w /work assurance-build'
eval $RUN python -m evidence_review.cli generate       # seeded synthetic cases and labels
eval $RUN python -m evidence_review.cli run            # run the 3 variants and evaluate (mock mode)
eval $RUN python scripts/build_p5_report.py            # PDF and HTML evaluation report
eval $RUN python scripts/build_p5_queue.py             # reviewer queue page
eval $RUN python -m pytest tests -q                    # unit and reconciliation tests
```
Open `public/p5/reviewer-queue.html` in a browser (works from a file or any static host).

## Layout
`src/evidence_review/` rules.py (deterministic library), gen_cases.py (seeded generator), mock_model.py, schema.py, adapter.py (mock and gated live adapter), pipeline.py, evaluate.py, cli.py. `prompts/evidence-review-v1.0.txt`, `schemas/ai-review-output.schema.json`. Tests in `tests/ai_review/`.

## Architecture
snippet + structured record -> deterministic rules -> (optional) AI step returns schema-validated JSON -> combine -> reviewer queue -> human decision (stored locally) -> audit log. Invalid model output becomes "Needs review".

## Live model mode (off by default, untested against a real API)
Requires `AI_MODE=live`, `ANTHROPIC_API_KEY` and `AI_MODEL` in `.env`, and `--allow-external`. Run `python -m evidence_review.cli dry-run` first: it prints exactly what would leave the machine (case id, evidence id, control id, declared type and snippet text; never the structured record or ground truth). Costs were not estimated because no live run has been made. Treat any live result as a new evaluation with its own model version and date.

## What the results mean
See `reports/pdf/ai-evidence-review-evaluation-report.pdf`. Short version: the harness works and the assisted pipeline detected more seeded exceptions than rules alone on this synthetic corpus; this is not evidence that AI improves audit quality.

Protocols, change log and data dictionary: `docs/ai-review-protocols.md`, `docs/ai-review-change-log.md`, `docs/ai-review-data-dictionary.md`.