Where we are
Two years ago, the best large language models scored below 50% on FinQA - a benchmark testing numerical reasoning over earnings documents that most finance graduates handle without difficulty. Today, frontier models score above 80% on the same tasks. The improvement has been fast, relatively linear, and largely driven by better base training rather than task-specific fine-tuning.
That progress obscures a more uncomfortable reality: on the tasks that actually matter in financial operations - verifying whether a transaction was charged the correct interchange fee, reconciling scheme rules across card networks, spotting the line in a waterfall structure where funds were misdirected - performance is still poor. Not 50% poor. More like 52-73% poor, depending on which frontier model you ask and how hard the task is.
What frontier models can do
The honest answer is: quite a lot, as long as you are asking them to retrieve and summarise rather than verify and reconcile. On FinanceBench, which tests question answering over real earnings documents, Claude Opus 4.7 scores 89.2%. Ask it what Apple's revenue was in Q3 2025 and it will tell you correctly almost nine times out of ten.
Models are similarly strong on tasks that require multi-step arithmetic over clearly structured tables - FinQA, where the best models now score above 80%, tests exactly this. The reasoning chain is visible, the tables are clean, and the questions have been designed with human annotators in mind. These are good benchmarks. They just do not test the hardest part of financial reasoning.
Where they still fail
Three failure modes show up consistently across our evaluations. The first is rule application under ambiguity. When a transaction spans multiple scheme rule categories - a cross-border credit card purchase at a merchant whose MCC sits on a category boundary - models struggle to apply the correct rule hierarchy. They know the rules individually. They fail at adjudicating between them.
The second failure mode is error propagation in multi-step reasoning. On Level 3 of the PAVE benchmark, tasks require models to identify fee leakage across scheme boundaries - which means correctly computing fees for dozens of transactions, comparing them to what was actually charged, and identifying the discrepancy. An error in step two corrupts steps three through twelve. The best model scores 52% on this task class.
The third is confidence calibration. Models that get a calculation wrong rarely flag it as uncertain. They produce a wrong answer with the same fluency as a correct one. In financial contexts where a downstream decision depends on the answer, this is not a minor issue.
The verification gap
The deepest problem is not performance on any particular task. It is the absence of a mechanism for self-verification. A human analyst who computes an interchange fee can check their work against the scheme rulebook. A language model cannot meaningfully audit its own output - it can regenerate an answer from a different angle, but it lacks the formal grounding to know whether either answer is correct.
This is what makes adversarial consensus verification architectures interesting. Rather than asking a single model to produce an answer, you ask multiple models to produce and critique answers, and you use disagreement as a signal of uncertainty. When models trained on different data distributions disagree about a financial calculation, that disagreement is informative. It is not a perfect substitute for formal verification - but it is a practical approximation that improves reliability without requiring symbolic AI.
The PAVE benchmark was designed to stress-test exactly this property. Level 4 tasks - strategic impact modelling - are designed to have answers that require multi-step reasoning where intermediate results are verifiable. Models that produce the right final answer via wrong intermediate steps score zero on these tasks. This is intentional. We care about reasoning chains, not just outputs.
What comes next
The near-term trajectory is clear: models will continue improving on document QA and structured numerical reasoning as base training data expands and reasoning capabilities compound. The harder question is whether verification - deterministic correctness on tasks with formal ground truth - will follow the same curve or require fundamentally different approaches.
Our tentative view is that it will require both. Better base models will raise the floor. But reaching the reliability threshold that financial institutions actually need - not 73%, but 99.x% - will likely require hybrid architectures that combine LLM reasoning with formal verification components. The benchmark exists to measure progress toward that ceiling, not just along the easy part of the curve.