Your Agent Evals Are Testing the Wrong Failures

Here is a pattern worth noticing in your own work. You build an agent. You write evals for it, because you are responsible. The evals pass. Then the agent does something stupid in production that none of your evals covered, you fix it, and you move on without adding a test for the thing that actually broke.

Six weeks later something similar breaks again. Your eval suite, meanwhile, has been passing continuously the entire time.

The problem is not that you wrote too few evals. It is that eval suites are written from imagination and production failures come from reality, and those two sets overlap far less than anyone expects.

Why imagined failures are the wrong ones

When you sit down to write evals, you are asking yourself: how could this go wrong? You are good at that question for the failure modes you already understand. You write cases for empty input, malformed JSON, the ambiguous instruction you argued about in review.

What you cannot write is a test for the failure you have not imagined yet. And with agents, the unimagined failures dominate, because an agent has a much larger space of things it can do than a function does. It can choose the wrong tool. It can loop. It can succeed at a subtly different task than the one you asked for. It can be confidently, fluently wrong in a way that passes every structural check you have.

Traditional testing handles this with regression tests: something breaks, you capture it, it never breaks again. That discipline is completely standard in normal software and almost entirely absent from agent development, because capturing an agent failure has historically been awkward. The failure is not a stack trace. It is a whole trajectory of decisions.

The trace is the test

The interesting move, and the one that Tracely is built around, is to treat the production trace itself as the artifact. When an agent fails, you already have the complete record: the input, the reasoning, the tool calls, the outputs at each step. That trace is a far better test case than anything you would have written by hand, because it is a thing that actually happened.

The workflow inverts. Instead of imagining failures and writing tests for them, you capture real failures and promote them into the suite. The suite grows in exactly the direction your application actually breaks, and it blocks the pull request if a regression reappears.

It is the same idea as a regression test in any other codebase. It just took a while to become practical for agents, because the tooling to capture and replay a full trajectory did not exist.

What this changes in practice

Your eval suite stops being aspirational

Most agent eval suites are written once, at the start, and then quietly stop reflecting the product. A trace-driven suite is updated by the act of debugging, which is work you were going to do anyway. The maintenance cost drops close to zero because capturing the case is a side effect of investigating the incident.

You get a real signal about whether you are improving

A suite of imagined cases that always passes tells you nothing. A suite built from things that genuinely broke gives you a meaningful number: of the failures this system has actually produced, how many are now prevented? That is a metric worth watching over time.

Prompt changes stop being terrifying

The reason people are scared to touch a working prompt is that they have no way to know what they broke. Prompts are global: a change to fix one behaviour can silently alter twenty others. With a trace suite, changing a prompt runs every real failure you have ever recorded. That is the difference between editing hopefully and editing with evidence.

What a captured failure actually looks like

An abstract argument for trace-driven testing is easy to nod along to and hard to act on, so it is worth being concrete about the shape of the thing.

Say you have an agent that answers questions about your documentation. A user asks something ambiguous. The agent picks a search tool, issues a query that returns nothing useful, does not try a second query, and confidently answers from its own general knowledge instead. The answer is fluent, plausible, and wrong. Nothing errored. Your output schema validated perfectly. Every eval you wrote still passes.

The trace of that interaction contains everything you need: the original question, the tool the agent selected, the query it constructed, the empty result, and the decision to answer anyway. As a test case, the assertion is not "produce this exact text". It is something closer to "when retrieval returns nothing relevant, the agent must either retry with a different query or say it does not know".

That is a property, and properties are what make replayed traces useful rather than flaky. You are not asserting that the agent walks the identical path. You are asserting that a specific bad behaviour, which you have now seen happen, does not happen again.

Notice also that you could not have written this test in advance in any useful way. You might have imagined "handle empty search results" as a category, but the specific failure, answering confidently from parametric knowledge rather than admitting ignorance, is the kind of thing you only learn by watching the system run.

How to start without adopting anything

You do not need a framework to get most of this benefit. The discipline is what matters, and you can start it this week.

1. Log complete trajectories, not just final outputs. Input, each tool call and its result, each intermediate decision, the final answer. If you only log the answer, you cannot reconstruct why it was wrong.
2. When something fails in production, save that trajectory to a file in your repository before you fix anything. This is the step everyone skips, and it is the one that matters.
3. Write the expected behaviour next to it, in whatever form you can check: an exact output, a property that must hold, or a rubric an LLM judge can apply.
4. Run the whole collection in CI on every change to prompts, tools, or model version. Especially model version.
5. When a case is genuinely obsolete because the product changed, delete it deliberately. Do not let it rot into a permanently skipped test.

That is a directory of JSON files and a test runner. The tooling makes it nicer, but the practice is what produces the benefit, and the practice is free.

The failure mode of this approach

It is worth being honest about where this goes wrong, because it is not a free win.

The first risk is overfitting. If your suite is nothing but captured failures, you are optimising against a list of past mistakes rather than against doing the job well. You still need a set of cases describing correct behaviour on ordinary inputs, and those still have to be written by hand.

The second is non-determinism. Replaying an agent trajectory is not like replaying a unit test. The model may take a different valid path this time. If your assertions are too strict, you get a flaky suite that everyone learns to ignore, which is worse than no suite. Assert on outcomes and properties, not on the exact sequence of steps.

The third is cost. Every eval run is inference you pay for. A suite of two hundred captured traces running on every commit gets expensive quickly. Most teams end up running a fast subset on each push and the full set nightly, which is a reasonable compromise.

Where this fits alongside your other tooling

Trace-driven regression testing is not a replacement for the observability tools you may already be running. It sits directly on top of them, and if you already have tracing wired up you are most of the way there.

Observability answers "what is happening in production right now". It shows you latency, cost, token usage, and lets you inspect individual runs when something looks off. What it generally does not do is turn any of that into a gate. You look at a bad trace, you understand it, you fix the prompt, and the trace stays in the dashboard as history rather than becoming a rule the system must satisfy from now on.

The addition is small but changes the character of the work: a step where a specific trace is promoted from "something I looked at" to "something CI enforces". If you already emit structured traces, the missing piece is a promote command and a runner, not a new platform.

It also composes with the ordinary discipline of pinning model versions. A large share of agent regressions arrive not because you changed anything, but because the model underneath you did. A suite of real captured failures is the cheapest early warning you can have when a provider ships a silent update, which is a good argument for running the full set on a schedule rather than only on your own commits.

Practical takeaways

• Your eval suite tests failures you imagined. Production produces failures you did not
• Log full trajectories, not just final outputs. You cannot debug what you did not record
• Save the failing trace before you fix the bug. This single habit is most of the value
• Run every captured failure on any change to prompts, tools, or model version
• Assert on outcomes and properties, not exact step sequences, or you will build a flaky suite
• Keep some hand-written cases for correct behaviour so you do not overfit to past mistakes
• Split the suite: fast subset per commit, full set nightly, to keep inference costs sane

The underlying point

Agent development still largely lacks the feedback loops that normal software engineering takes for granted. We have version control and CI, but the thing being tested is probabilistic, and most teams responded by testing less rigorously rather than differently.

Treating production traces as the source of your test suite is the closest thing available to the regression discipline that made the rest of software tractable. It does not require you to predict how your agent will fail, which is fortunate, because you cannot.