Proving It: Evaluating Enterprise AI Without a Trusted Answer Key

Fourth in our series on evaluating AI agents. The earlier posts:

1. Evals First, Agents Second
2. A Practical Framework for Evaluating AI Agents
3. Your LLM Judge Is Just One More Model to Test

A Quiet Assumption

Before an enterprise customer trusts an AI agent, they want proof that it is accurate. Most evaluations rest on a quiet assumption about how that proof works: the customer’s answer key is the standard, and the vendor’s job is to match it.

The assumption is reasonable. The customer knows their domain best. They own their documentation. They define what “correct” means for their products.

But in the real engagements we run, the answer key is rarely that clean. Take two recent case studies:

1. The first customer arrived with 600+ queries and no ground truth at all. Just queries, and a request to prove our agent was accurate.
2. The second arrived with 800+ queries and what looked like a solid answer key. The key itself turned out to contain significant errors.

The first taught us how to build an evaluation from nothing. The second taught us to handle scenarios where the answer key itself is not reliable. We look at both scenarios in more detail in the remainder of the post.

Building Ground Truth From Zero

No expected answers, no labels, no subject-matter-expert-validated dataset. Just 600+ queries.

That is simply how a lot of real engagements begin. Building and maintaining a trusted answer key at that scale is genuinely expensive, and a domain expert’s time is usually better spent on the product than on hand-labeling hundreds of rows.

Knowing that this is the common case, we built ourselves a workflow for starting from ground zero. The first task it has to address is the obvious one: generating ground truth.

The tempting move is to ask another large language model (LLM) to generate the expected answers and call that the dataset. Fast, tidy, and very risky. If the ground truth is built from model output, it runs into the same risks of hallucinations, blind spots, and confidently incorrect responses we are trying to prevent in the target agent.

Day one: the ask, and the shortcut we turned down.

So we did the slower but correct thing. Our three-part Dataset Scaffolding Workflow, explained next.

Step 1. Keep the retrieval trail

The customer’s documentation was already ingested into our knowledge graph. When a query came in, the agent pulled evidence from every source it had access to, and we stored each retrieved chunk alongside the answer. Every claim in the answer cited the chunk it relied on.

That preserved trail is what makes unsupervised evaluation possible at all. With it, an answer can be graded against the evidence that produced it instead of against a paragraph someone hopes is correct.

Each (query, answer, evidence) triple then gets three checks:

1. Faithfulness. Does every factual claim trace back to retrieved evidence?
2. Completeness. Does the answer cover what was asked, without filling gaps with speculation?
3. Relevance. Did it answer the user’s query, or an adjacent one?

We covered how we design and score checks like these in A Practical Framework for Evaluating AI Agents.

Step 2. Make the answer key a checklist, not a paragraph

For each query we built the key from the customer’s own source material, but not as a single golden paragraph. It was an assertion-level dataset, split into three buckets:

• Must-have. The required facts a correct answer cannot miss: the specific product identifier, exact rating, lifecycle status, supported feature, or relevant document reference.
• Good-to-have. Not required for correctness, but they make the answer more useful: related documentation, compatible alternatives, implementation context, or a companion file that gives the user a better next step.
• Do NOT accept. The trapdoors. The official-looking wrong claims we explicitly wanted to catch: fabricated identifiers, wrong specifications, unsupported dates, deprecated product claims, or saying something is not publicly available when the source material publishes it.

That last bucket mattered more than we expected. In enterprise technical domains, hallucinations rarely look like obviously visible errors. They look polished. A model can include the right product name and still sneak in a fake compatible part. It can cite the right product family and still attach the wrong rating. It can politely say “contact support” for a document sitting publicly in the source material. Positive facts alone do not catch any of that. The evaluation needs a veto lane.

Every assertion is checked against the customer’s own sources: what must be there, and what must not slip in.

Every assertion had to trace back to primary source material before it entered the dataset. If a claim could not be tied to a specific published source, it did not become ground truth. We also kept the assertion set intentionally small. The goal was not to grade every sentence, only the facts that would actually change whether an answer was correct, relevant, and complete.

A simplified example:

Query: Is the technical file for Product B publicly available?

Must-have: State whether the file is publicly available, based on the source material.

Good-to-have: Point to the document, file reference, or product page where the user can verify it.

Do NOT accept: Do not claim the file is unavailable, hidden, or only accessible through support if the source material publishes it.

Step 3. Treat a low score as a question, not a verdict

This structure made failures precise. When an answer scored poorly, we could ask something better than “was it wrong?” Instead:

• Did it miss a must-have fact?
• Include a forbidden claim?
• Answer from unsupported evidence?
• Did retrieval fail?
• Or was our own assertion too strict or too lax?

So we treated the judge’s scores as the start of an investigation rather than the finding. In the early runs we reopened every low-scoring case against the underlying source material and tuned the judge and the assertions until the scores tracked what a careful human reviewer would say. That is also how we caught the cases where the answer key, not the agent, was the thing that needed fixing.

The judge starts the process. The human audit keeps it honest.

Reopening those cases turned up four recurring findings, and only one of them was our agent’s fault:

• Over-cautious negative guards. A “Do NOT accept” assertion forbidding a claim the source material actually supported.
• Sources we had written off. Documents we had recorded as unavailable that were in fact published.
• Unindexed answers. The right answer sitting in a document we had not ingested yet, which is a retrieval gap on our side.
• Holes in the customer’s knowledge base. Products that had quietly gone legacy and now existed nowhere our retrieval could reach. Specifications that lived in someone’s inbox but not in any indexed document. Naming inconsistencies between product lines that even the customer’s own experts had stopped noticing.

The first three sent us back to fix the dataset or our own pipeline and re-run. The fourth was a surprisingly common pattern. Those were not bugs in our agent at all. They were holes in the customer’s documentation, made visible by an agent that could not find an answer where one should exist.

What our dataset scaffolding workflow achieves

The customer ended up with a structured ground truth dataset they did not have at the start, plus a list of gaps to fix at the source. We ended up with a regression suite that catches failures the moment our pipeline, the content, or model behavior changes.

When the Answer Key Is Wrong

The second customer’s 800+ queries came with an answer key, which should have made evaluation easy. Right?

To our surprise, our agent’s initial result came back at around 70% correct, much lower than we expected and with seemingly embarrassing errors. There were two ways forward.

• Chase the number. Tune prompts, fine-tune the model, rework retrieval and the pipeline until the score climbs. The benchmark game.
• Dig into each failure and find out what is actually happening. The knowledge game.

We chose the second. After spot-checking a handful of failures, the pattern looked strange. The customer’s own documentation, retrieved by our agent, contradicted the expected answer in case after case.

The patterns we found

For each failure we asked three questions. Which document and which section did the agent retrieve? Did that content support, contradict, or have nothing to do with the expected answer’s claims? And if it contradicted, was that a retrieval error on our side or an error in the expected answer itself?

In the majority of cases the retrieved content contradicted the expected answer, and the contradicting evidence came from the customer’s own official documentation. Tracing a chunk back to its source URL landed on content the customer had published about their own product.

The customer had generated their answer key with an LLM. Call it the customer LLM, to keep it distinct from the model inside our agent. Its mistakes were not randomly distributed. They came in three clusters.

• Template hallucination. Several queries on the same topic shared the same incorrect template in their expected answers, with nearly identical wording. A single hallucination by the customer LLM had been replicated across many queries.
• Cross-product entity confusion. Expected answers cited entity names that did not exist in the customer’s documentation. The correct names were in the same documents, just in slightly different forms. The customer LLM appeared to have pulled naming conventions from a different product line and misapplied them.
• Misreferenced citations. Expected answers cited “according to document X, section Y” but pointed at sections for unrelated products with similar names. The customer’s own documents, misreferenced.

To check that this was not just our own bias, we re-ran a focused set of these failures through a blind LLM judge: the customer’s expected answer and our agent’s answer, unlabeled, with the customer’s documentation as reference, scored on completeness, factual correctness, and reference quality. Ours was rated higher in the majority of them. That is not a “we won” result. It is evidence that the ground truth itself was imperfect, and was marking correct answers as wrong.

The cases we lost mattered more

• Retrieval gap. Our agent confidently said a feature was not supported. A section of the customer’s documentation that we had never ingested described exactly how that feature worked.
• Hallucination. Our agent invented a spec that was not in the source, and in another case placed wrong details next to correct ones.

Both went straight onto our work list. The process does not only refine the customer’s dataset. It improves our agent in parallel. It is bidirectional work.

Why This Usually Goes Unreported

Hand-writing evaluation queries with subject-matter experts can take months, so it’s tempting for enterprise teams to generate query and expected-answer pairs from their documentation with an LLM instead. Productivity feels high with the ability to generate thousands of query-response pairs within a day. The cost of this shows up when the expected answers carry the failure modes of the model that wrote them, in the form of an expected answer that has drifted away from the source material it was supposed to be derived from, while still reading as authoritative. The timeline delays caused by this can be significant, so we caution our customer teams from pursuing this approach.

The awkward part is on us. It is never comfortable to tell a customer “your expected answer is wrong.” Nobody enjoys being corrected, and from the outside it can look like the vendor is gaming the evaluation to improve its own numbers. The easy path is to stay quiet: tune prompts until the score passes and never mention the mismatches.

That silence costs the customer more than it costs us. Quality problems in their documentation stay invisible, because the one process that could have surfaced them was optimized to paper over them instead.

The Stack That Made This Possible

This kind of analysis is not possible with plain retrieval-augmented generation (RAG). You need to be able to trace where an answer came from, why the agent reasoned the way it did, and how that lines up with the customer’s source material.

• Precision retrieval. Not “the document was used in generation” but “this specific source supports this specific claim”, at the sentence level.
• Per-claim citation. Each claim carries a deeplink to the exact location in the source. Without it you cannot compare expected answers against retrieved evidence to find mismatches.
• Knowledge tracing. The agent’s reasoning path has to be inspectable. Why this answer? Which evidence was weighted?
• Dataset infrastructure with an LLM judge. Something that can put the expected answer and the agent’s answer side by side and have a third model compare them blindly, so your analysis is not just self-serving. The judge needs tuning of its own, which is the subject of Your LLM Judge Is Just One More Model to Test.

With those four in place, an AI agent stops being a query-to-answer tool. It becomes a mirror for the enterprise knowledge system, and a working mirror shows the customer the parts of their knowledge base they could not see themselves.

From 70% to 95%

Here is what the investigation did to the number.

Each time our retrieval surfaced documentation that contradicted an expected answer, we brought it back to the customer with the source attached: this expected answer does not look right, and here is the page that says so.

Saying that takes a particular kind of confidence. Not confidence that we are always right, but confidence in the pipeline behind the answer: precision retrieval, a citation on every claim, a traceable reasoning path. When you can show the exact source for what you said, “we think your answer key is wrong here” stops being a defensive excuse and becomes a claim the customer can check in seconds.

As the flawed expected answers were corrected at the source, the same evaluation that had scored us at 70% rose to about 95%. Our agent’s answers had barely changed. The answer key had.

Bidirectional Evaluation

What we have come to believe is that enterprise AI evaluation has to move from a one-way model to a two-way model.

One-way. The customer sets the standard and the vendor matches it. Before synthetic eval data, this worked well: experts wrote the queries and answers themselves, so ground-truth quality was more or less guaranteed.

Two-way. The vendor surfaces where the customer’s expected answers have drifted from the customer’s own sources. The customer fixes the data at the source. The customer’s cases expose the vendor’s retrieval gaps and reasoning weaknesses. The vendor fixes its stack. Both sides improve the evaluation together.

What makes this work is a collaborative mindset in the pursuit of quality:

• Transparency from the vendor. Showing why the agent answered the way it did and which sources it consulted, so a disagreement is checkable rather than a matter of opinion.
• Capacity on the customer side. When we hand over a summary of the problems our evaluation found in their answer key, someone has to be staffed to act on it. In practice this is the real differentiator. No customer has argued with our findings. The ones who got the most out of the process were the ones with the people and the organizational will to work through the list and improve the dataset.

When we sat down with this customer and made the case directly, they were already thinking the same way. What we agreed on:

Improving agent quality is a priority, and we’ll commit to both allocating subject matter experts to review answer quality, plus fix inaccuracies and incompleteness in the source documentation.

The Next Chapter for Enterprise AI

As AI adoption picks up, being able to do high quality evaluations becomes a critical need. Through the experiences we shared in this post, we’ve only grown more enthusiastic about being a trusted partner to our customers. Evaluation is a first-class engineering problem, not a gate bolted onto the end of a project. Done carefully it works as a diagnostic, exposing content quality gaps, coverage holes, and model behavior we can act on. That is how the last stretch of quality gets closed, and how a customer moves from skepticism to a defensible baseline for what “good out of the box” actually means.

The real value of an AI agent may not be in passing a fixed quiz. It may be in helping raise the quality of the quiz itself.