The AI demos that impress people are the ones that generate fluent text. Ask for a company overview, a market section, a polished paragraph of investment rationale, and a modern language model produces something that reads remarkably well. It’s genuinely impressive, and it’s also the easy part. The hard part — the part that actually decides whether an AI tool can be trusted on a deal — is reading a single reliable number out of a messy financial document.
That asymmetry is counterintuitive, so it’s worth being precise about why it holds. Fluency is forgiving. A number is not.
Fluency degrades gracefully; numbers don’t
When a language model writes prose, small errors are absorbed. A slightly awkward sentence, a marginally off emphasis, a generic phrase — none of it breaks the output. The reader smooths over it, and the document is still useful. Text degrades gracefully: it gets a little worse without becoming wrong.
A number does the opposite. There is no “slightly wrong” EBITDA. The figure is either the one in the source document or it isn’t. A model that reads €4.2m as €4.7m hasn’t produced a marginally worse output; it has produced a false one, and a false number in a financial deliverable doesn’t degrade gracefully — it detonates. A misstated figure in an IM is a credibility event that travels to every other number in the document.
So the two tasks have fundamentally different error tolerances. Generating text is a domain where “mostly right” is useful. Reading financials is a domain where “mostly right” is dangerous, because the one wrong figure is indistinguishable, on the page, from the correct ones around it.
Why reading a financial document is genuinely hard
“Just extract the number” sounds mechanical. It isn’t, and the reasons are specific to financial documents.
The same figure appears in conflicting forms. Revenue for FY24 might appear in the management accounts, the audited statements, a board pack, and a teaser — and the four don’t agree, because one is pre-adjustment, one is post-, one is a forecast that was never trued up, and one is rounded. Extracting “revenue” isn’t reading a number; it’s knowing which number is authoritative and why. That’s judgment, and a model with no concept of source hierarchy will confidently grab whichever it saw first.
Scale and currency are ambiguous. A figure of “4,200” might be €4,200, €4.2m (the table is in thousands), or €4.2bn (a different table, different units). The unit is often declared once, in a header three pages away, or implied by context a human knows and a naive extractor doesn’t. Get the scale wrong and the error is 1000x, not marginal.
Labels drift and lie. “Other operating costs” in one year is “Sundry overheads” the next and is split into two lines the year after. A figure labeled “EBITDA” might be reported, adjusted, or management-adjusted, and the three differ by exactly the add-backs a buyer will fight over. Reading the number correctly requires understanding what the label means in context, not just matching on the word.
The source is hostile to extraction. Financial data in the mid-market arrives as scanned PDFs, broken Excel exports, management accounts assembled by hand, and the occasional figure that exists only in an email. Each format fails differently, and none of them is the clean, structured input a demo uses.
None of this is a text-generation problem. All of it is a reading problem, and reading — in this precise, source-aware, unit-aware, label-aware sense — is where the difficulty actually lives.
Why “the LLM said so” isn’t a standard
Here is the crux. A language model, asked for a number, will give one. It will be fluent, confident, and formatted correctly. None of those properties tell you whether it’s right. The model’s job is to produce plausible output, and a plausible wrong number is the single most dangerous thing an AI can hand a deal team, precisely because it looks exactly like a correct one.
This is why “the model extracted it” can’t be the standard for a number that goes into a deliverable. Fluency is not accuracy. Confidence is not correctness. In a context where a wrong figure is a credibility event, the only acceptable standard is one where every number can be checked — traced back to the document, the page, the line it was read from, so a human can verify it in seconds rather than trust it on faith.
That’s the real dividing line between an AI tool that belongs near a deal and one that doesn’t. Not how good the prose is. Whether every number it produces can be walked back to its source.
What this means for evaluating AI in finance
The implication for anyone assessing AI tools for deal work is to test the hard thing, not the easy one. A demo that generates a beautiful company overview is testing fluency, which modern models have largely solved. The question that actually matters is what happens when you point the tool at a messy, real data room and ask it for a number:
- Can it show you the exact source — document, page, line — behind every figure?
- Does it know which of four conflicting versions of “revenue” is authoritative, and can it tell you why?
- Does it get scale and currency right when the units are declared three pages away?
- When it isn’t sure, does it flag the ambiguity, or does it produce a confident guess?
A tool built for finance treats reading as the hard problem it is: provenance-first, source-traced, with the number always tied to where it came from. A tool that treats extraction as a solved afterthought — fluent output, no traceability — is impressive in a demo and dangerous on a deal.
The fluent text was always going to be easy. Reading the numbers correctly, and proving you read them right, is the part worth building for.
NaS_OS is built provenance-first: every figure it reads from a data room is traced to its source document, page, and line, so the number can be verified rather than trusted on faith. If you want to test that on your own documents, apply for access.