Why Forensic Discovery Reveals Model Limitations

Why forensic discovery tasks reveal what frontier models still cannot do

Document QA benchmarks make frontier models look better than they are. Forensic discovery - finding what is missing, inconsistent, or deliberately obscured - is a different task. We explain why, and what happens when you test for it.

FinanceBenchmark Research

financebenchmark.ai

·Published Jun 18, 2026·Updated Jun 18, 2026·7 min read

Document QA vs forensic reasoning

There is a meaningful difference between asking a model what a document says and asking it what a document is hiding. Financial QA benchmarks - FinanceBench, ConvFinQA - test the former. You hand the model a 10-K and ask it to find the revenue figure. The answer is there. The model just has to retrieve it.

Forensic tasks are different. You hand the model a set of documents and ask it to identify where the story does not add up. The answer is not in the documents - it is in the gap between them. This requires not just retrieval, but hypothesis generation, cross-document reconciliation, and the ability to notice absence.

What forensic tasks look like

On FinAuditing, the hardest task class requires models to read an audit report, the underlying financial statements, and the notes to those statements, and identify where the three documents are inconsistent. Human forensic accountants spend careers developing the pattern recognition for this. Frontier models score 62–71% - well above chance, but far below the reliability threshold for any real deployment.

The failure mode is predictable: models find inconsistencies that are explicitly flagged somewhere in the document set, and miss inconsistencies that require synthesising information across documents that were never intended to be read together.

How models perform

GPT-5.5 leads on FinAuditing at 71.0%, followed by Claude Opus 4.7 at 68.4% and Gemini 3.1 Pro at 65.1%. The gap between models is smaller on forensic tasks than on document QA - which suggests that whatever capability separates the top models on retrieval tasks matters less when the task requires synthesis. This is counterintuitive but consistent across our evaluations.

Implications for deployment

Financial institutions deploying AI for document review should test specifically on forensic task classes before drawing conclusions from document QA benchmarks. A model that scores 89% on FinanceBench may score 65% on tasks that require it to notice what is missing. These are not the same capability, and they do not correlate as strongly as practitioners assume.