If you can't measure it, you bought a vibe

There’s a pattern we keep seeing. A team pilots an LLM system — document extraction, a RAG assistant, an agent that handles some workflow. The demo is genuinely impressive. Everyone in the room watches it nail three examples, someone says “ship it,” and it ships.

Six months later nobody trusts it. Not because it collapsed — because it quietly got things wrong on the cases that actually mattered, and nobody could say how often, or whether it was getting better or worse. Ask “how accurate is it?” and the answer is an adjective: pretty good, mostly right, surprisingly solid. Adjectives are what you have when you never built a way to measure. You didn’t buy a system. You bought a vibe.

The demo is the easy part

A demo shows the system working on inputs someone chose. Production is the system working on inputs nobody chose — the scanned PDF with a coffee stain, the contract with three amendments, the question phrased the way real users phrase things. LLMs fail differently from normal software: they don’t crash, they answer confidently and wrong. Without measurement, that failure mode is invisible right up until it’s expensive.

This is also the honest test for vendors. Ask what accuracy they get on data like yours, field by field. A number, with a methodology behind it, is a real answer. “It’s very accurate” is a demo with a sales team.

The eval harness is the product decision

The uncomfortable reframe: deciding whether an AI system is production-ready is deciding what to measure and what number is acceptable. Before agents, extraction, or retrieval go anywhere near production, the harness comes first:

• A golden set of real cases — including the ugly ones. Fifty representative documents or queries, with the correct answers established by a human, beat five hundred clean ones.
• Accuracy on the fields that matter. Not “did it produce output” but “was the royalty rate right, the expiry date right, the claim number right” — scored per field, because an extraction that’s 95% right overall and 60% right on the one field you make decisions from is a 60% system.
• A failure-mode taxonomy. Wrong answers aren’t one category. Hallucinated values, missed fields, format drift, and confidently-answered out-of-scope questions each need different fixes. If you can’t name the failure modes, you can’t reduce them.
• A human-review lane for low-confidence output. The system should know what it doesn’t know, and route those cases to a person instead of guessing.
• Regression checks on every change. Models get deprecated, prompts get “improved,” vendors update endpoints. Each of those is a code change to a system with no compiler. The golden set is what tells you the new version didn’t break the old cases.

The honest caveat

Evals cost real effort — building the golden set means a person with judgment labelling real cases, and maintaining it isn’t free. Match the rigor to the consequence. An internal tool that drafts first-pass summaries someone always reads before using? A lightweight spot-check regime is proportionate. A system extracting numbers that feed a financial model, a regulatory filing, or a public map? That’s where “we eyeballed it and it seemed fine” stops being a methodology.

The question to ask isn’t “should we build evals” but “what happens when this is wrong, and who finds out.”

Where we sit

We build AI document systems and agentic workflows, and we run an agentic production system inside our own studio — which is exactly why we’re insistent on this. Every extraction pipeline we ship comes with its golden set, its accuracy numbers, and its review workflow, because a system we can’t measure is a system we won’t stand behind. If you’re evaluating an AI system — yours or a vendor’s — and want the question “how accurate is it?” to get a number, see our capabilities or get in touch.