A Practical Framework for Evaluating AI Agents

The challenge with evaluating agents

Early agent development can rely on manual testing for a while. You try a few representative tasks, dogfood the system, fix the obvious gaps, and keep moving. That works when the surface area is small and the behavior you care about is still easy to inspect by hand.

The problem shows up as the agent starts handling more workflows and more real users. A change can make one part better and quietly weaken another and without a structured eval suite, it is hard to tell whether a reported issue is a real regression, an isolated bad case, a data problem, or just noise.

At Rapidflare, we build domain-specific AI agents for technical sales and support in electronics. Our agents answer product questions, compare parts, generate recommendations, support proposal workflows, and reason over dense technical documents.

Evaluating systems like this is non-trivial because we need to know whether an answer is factually correct, grounded in the right sources, useful to the user, and still reliable as models, prompts, and data change underneath it.

In this post, we walk through the framework we built to evaluate these systems more systematically: how we structure evals, design cases, choose evaluators, and turn failures into something we can act on.

The basic structure of an eval

Before going into the details, it helps to clarify the basic shape of an evaluation. Every eval has three parts:

1. A dataset: what you ask, and what counts as success.
2. A target: the thing you’re testing.
3. Evaluators: how you score the output.

Our framework builds on that structure. Here are the main components:

A test case is one scenario with defined inputs and expected behavior. In our datasets, each case includes the user query, the assertions we expect the answer to satisfy, and metadata such as traits, distractors, and the goal of the case.

A case spec is the designed part of a test case: the assertions the answer must satisfy, plus the traits, distractors, and goal that describe what the case tests and what makes it tricky. Each of these fields is covered in detail later in the post.

A dataset is a collection of test cases. It also declares which evaluators should run and, optionally, which configurations the dataset should be tested against.

A target is the system being evaluated. For us, that is usually an agent or endpoint running the same code path we use in production.

A runner executes the dataset against the target. It sends each case to the target, collects the output, applies the evaluators, and passes the results to the reporting layer.

Evaluators score the output across different aspects: correctness, relevance, clarity, tool use and so on. The framework ships with common evaluators, and if a dataset needs some specific behavior evaluated, we can define a custom evaluator and plug it into the same flow.

A trace is the full record of what happened during a case: the input, the output, tool calls, retrieved context, intermediate steps, and evaluator results.

Reports are how we present the results after the run finishes. In our case, we use LangSmith as the reporting layer, so each run has per-case scores, evaluator outputs, and traces in one place.

The last piece is the run matrix. It lets us run the same dataset across different configurations, such as provider, model, or thinking budget, and compare the outputs side by side.

For example, a small matrix might look like this:

run_matrix:
  - provider: ANTHROPIC
    model: claude-sonnet-4-6
    thinking_budget: adaptive
  - provider: OPENAI
    model: gpt-5.5
    thinking_budget: high

The dataset is where the design happens

Where cases come from

It is important for your evaluation dataset to be representative of the data you receive in actual production traffic. That means sourcing cases from production, but not just randomly sampling them. You also need to be intentional about which conversations you pick. For example, look at high-signal conversations: conversations with feedback (positive or negative), conversations where you can see user frustration, handoffs or escalations, and longer threads where the user keeps drilling down.

Anthropic makes the same point in its guidance for agent evals:

If you’re already in production, look at your bug tracker and support queue. Converting user-reported failures into test cases ensures your suite reflects actual usage; prioritizing by user impact helps you invest effort where it counts.

Apart from using production traffic for your dataset, you can also use synthetic test case generation for greater coverage of testcases.

However, synthetic dataset creation should be used with caution. You do not want to over-engineer your agent for an edge case you created if that behavior never actually happens in production. In the beginning, it is better to start with organic production data and keep humans close to that process. Reviewing the data by hand helps build intuition for what the eval suite is actually measuring.

Assertions, not reference answers

Reference answers sound like the obvious way to grade an eval, but there is a catch. A good answer often needs to mention several facts, constraints, tradeoffs, and caveats. If both the reference answer and the model answer are long and fact-packed, the judge can lose focus. It has to decide which facts matter most, which missing details are acceptable, and whether the answer is directionally correct or actually wrong.

That is why we grade against assertions instead: short, checkable rules that make the important parts of the response explicit.

outputs:
  assertions:
    - "Recommends at least one laptop with 16GB RAM or more under $800"
    - "Flags that the bestselling model in this range has only 8GB"
    - "Bonus: Mentions the upgradeability of the RAM"

Each bullet is meant to isolate one part of the expected behavior. On the flip side, that still leaves room for ambiguity. Plain English can mean different things to different judges. Is a “nice to have” required? If we say “recommends A or B,” does recommending only B count? To avoid that kind of confusion, we standardized a few patterns and tell the judge how to read them:

How the assertion is written	How the judge must read it
Bonus: mentions RAM upgradeability	Optional. Missing it cannot lower the score
Recommends model A OR model B	Either one passes
If the user asks about gaming, mentions the GPU	Passes if the user never asked, or if they asked and the GPU was mentioned
Asks about budget before recommending	Passes if the agent asked, or if it presented options with prices labeled

Once the judge knows how to read the assertions, we also give it a small grading scale:

Scoring (use the full 0.0-1.0 range; do not collapse to 0/1):
- 1.0 = Fully satisfies the reference; no contradictions, no fabrications
- 0.7 = Primary substance correct; one secondary detail missed
- 0.5 = Primary correct but with a partial offset
- 0.3 = Primary missed, but a plausible alternative offered without
        contradicting the reference
- 0.0 = Factual contradiction, fabrication, or refusal

The important part is to have a grading rubric that is anchored to the assertions. That is how we design our judge rubric: each score has to map back to what the answer did or did not satisfy. The more ambiguity you leave in the rubric, the more noise you introduce into the evaluation.

Traits, distractors, goals

At this point, we know where the test cases come from and how to define the evaluation criteria. But there is one more question worth asking: why does this test case exist in the first place, and why is it a good candidate for the dataset?

You might have the answer in your head: this case tests retrieval, or constraint handling, or multi-turn memory. That intuition is useful, but it is much more valuable when it is formalized into your test case.

We started by adding metadata fields to our dataset, beginning with traits.

Traits are the capability you’re testing. Here are a few of ours:

Trait	What it tests
spec_lookup	Find one grounded fact
numeric_grounding	The answer hinges on a specific number; a wrong number is worse than a refusal
constraint_satisfaction	Multi-part ask; every qualifier must hold
disambiguation	Ambiguous request; agent must clarify before committing
refusal	The correct behavior is to decline
premise_correction	The question rests on a false premise; agent must push back
anaphora_resolution	multi-turn reference carrying

Distractors describe what makes the case tricky. Inspired by Chroma’s generative benchmarking work, we tag the specific trap in the case, such as a confusing product name, an outdated spec, or a number that looks right but violates the user’s constraint.

Distractor	The trap
numeric_trap	Multiple plausible numbers; only one satisfies the constraint
confusable_product	Two products with nearly identical names
stale_vs_current	Old and new spec for the same property, both retrievable
lexical_overlap	Keywords match the wrong document

Goal states what the case is meant to verify. It should describe the behavior being tested and the regression we want this case to catch.

One case, fully designed

This is what a fully designed test case looks like in our dataset.

- id: catalog-rec-024
  inputs:
    query: "Suggest a laptop under $800 with at least 16GB RAM and a battery
            rated for 10+ hours."
  outputs:
    assertions:
      - "Recommends at least one model meeting all three constraints"
      - "Flags that the popular default pick in this range has 8GB, not 16GB"
      - "Bonus: Notes which models have user-upgradeable RAM"
  metadata:
    traits: [recommendation_fitment, constraint_satisfaction, numeric_grounding]
    distractors: [numeric_trap]
    goal: "Agent must not cherry-pick the popular product that violates a stated constraint"
    source: customer

One evaluator per dimension

You do not want one uber LLM judge doing all the grading. It is too easy for a single judge to mix concerns: correctness, clarity, relevance, tool use, and other dimensions all get blended into one score.

We found it more useful to keep evaluators focused. Each evaluator looks at one specific aspect of the output, so the score tells you what actually passed or failed:

• Correctness: did it satisfy the assertions?
• Clarity: is it readable?
• Relevance: did it answer what was actually asked?
• and many more

This separation was useful in our own runs. In one class of cases, the agent produced a clear recommendation, but one of the suggested products violated a hard user constraint. With a single overall score, that failure was harder to interpret. With separate evaluators, clarity could pass while correctness failed, which made the issue easier to diagnose.

Apart from splitting the panel by dimension, there are a few other evaluator design choices we recommend:

Use LLM judges sparingly. If something can be checked deterministically with code, it usually should be. Format validation, exact-match comparisons, schema checks, and tool execution are good examples. Deterministic checks are cheaper, faster, and more consistent than LLM judges. Save LLM judges for cases where the evaluation genuinely involves subjectivity.

Use agreement for borderline judgments. The same answer can get slightly different scores across judge calls, especially when it is close to the pass/fail boundary. For example, when evaluating correctness, we run the judge three times independently and take the median score. In our runs, this improved the measured correctness pass rate by about five points by removing judge-variance failures.

What designed cases let you see

This is where the metadata becomes useful.

An overall pass rate is useful, but it is only a summary. If your suite moves from 82% to 85%, you still need to understand what improved. If it drops from 85% to 80%, you need to know where to investigate. Once every case has traits, distractors, goals, evaluator scores, and traces attached, the aggregate score becomes easier to interpret.

The first useful view is slicing by trait. Instead of only asking whether the suite passed, you can look at which capabilities are strong and which ones need work. For example, if spec_lookup is healthy but constraint_satisfaction is weak, the agent may be finding the right facts but not applying all of the user’s requirements correctly. That calls for a different fix than a retrieval failure.

After looking at aggregate results, we inspect the failed cases more closely. This can be done by a human reviewer or with another evaluator, but the key is to look beyond the final answer. For agentic systems, the path matters: what the agent retrieved, which tools it called, what values it relied on, and where the evaluator disagreed.

For this, we use LangSmith as our observability layer. It keeps the full trace for a failed case in one place, including the input, retrieved context, tool calls, intermediate steps, final answer, and evaluator feedback.

At the top level, most failures fall into one of three groups:

1. Agent issue: the prompt, tool use, reasoning path, or final verification needs work.
2. Data issue: the source data is missing, stale, incorrectly parsed, or wrong.
3. Test issue: the assertion, rubric, or judge is grading the wrong thing.

After routing the failure, we add a more specific label if it helps explain the pattern: retrieval miss, hallucinated claim, hard constraint violation, and so on.

Once failures are routed and labeled, you can look for patterns across the whole suite. For example, in the heatmap below, many recommendation_fitment failures come from retrieval misses. That is useful because it tells you the agent may not be seeing the right products in the first place, so the fix is likely closer to retrieval than response writing.

Putting it together

The value of evaluations is in how quickly it lets you iterate. There is real upfront effort: collecting the right cases, writing assertions, defining traits, choosing evaluators, and making the reporting useful. But once that foundation is in place, every new experiment becomes easier to interpret.

For our team, this paid off quickly. With the framework in place, we evaluated and launched a new agent across 3 customers in about a month. It also gave us the confidence to replace a frontier model that cost roughly 3x with a cheaper one, because the evals showed the cheaper model was not only faster but gave better, more readable answers. Without the suite, that swap would have been a leap of faith. With it, it was a routine decision backed by data.

There is a quieter benefit too. Customers notice how thoroughly we evaluate, and walking them through what we test before a change ships has become a trust-building exercise in its own right.

That matters because evals are not just a way to measure quality after the fact. They help you make changes faster, understand what changed, and explain that work clearly to the people who depend on the agent.

In this post, we covered the basics: evals where you have a query and a set of ground truths to grade the answer against. That is a useful starting point, but it is not the whole system you need as agents become more complex.

As the agent starts taking more steps, calling more tools, and making more judgment calls, the evaluation harness has to grow with it. That includes trajectory evals that look at the path the agent took, tool-calling evals that check whether the right tools were used correctly, calibration evals for judgment-heavy cases, and evals for cases where there is no clean ground truth or assertion to grade against.

We will cover more of those patterns in upcoming posts, including how we run these suites efficiently as regression tests on every change. AI agent evaluation is still changing quickly, and we will keep sharing what we learn as we build.

---