A developer on r/ChatGPTCoding asked a question this week that every agent user runs into sooner or later. They run Codex workers that no longer ask before every command, so they stopped watching each step. That made one question matter more than anything: how do you know a task is really done?
Their first answer is the best one-line rule we have seen on the subject: done is an artifact, not a sentence. A code task is not done when the agent says the tests pass. It is done when you can see the real test run.
This post builds on that rule with three other sources from the same week: a hands-on tutorial on evidence-driven development, a write-up from the team behind Ramp Stack on grading agents end to end, and Tiden, a new tool that checks agent work against what you asked for rather than against the session that wrote it. Below: seven practices, from the cheapest to the most structured, and a minimal setup a solo builder can adopt this week.
Why "the tests pass" is not proof
When a coding agent writes a feature and its tests in the same session, the tests share the agent's understanding of the task. If the agent misread what you wanted, the tests encode the same misreading, and they pass. Tiden's launch puts it bluntly: most checks run inside the session that wrote the code, so the tests agree with it by construction.
There is a second gap. An agent's summary is a claim about the work, written by the thing that did the work. It can be accurate. It can also describe tests that never ran, a scenario that executed zero times, or a narrower promise than the one you asked for. None of that is malice. It is what happens when the only witness is the worker.
Every practice below closes one of those gaps.
1. Make done an artifact, not a sentence
The simplest change: stop accepting "done" as text. Ask for something you can inspect: the raw output of the test run, the command that produced it, and the files it touched. If the agent ran a suite, you want the runner's own output, not a paraphrase of it.
What counts as an artifact depends on the task. For a bug fix, it is a test that failed before the change and passes after, with both runs shown. For a refactor, it is the full suite before and after, with the same number of tests collected. For a UI change, it is a screenshot of the changed screen. For a data script, it is the output file and a row count. In every case you can check it without asking the agent anything.
The evidence-driven development tutorial frames the same idea as a principle: the coding agent can help at every step, but the evidence must stay inspectable without trusting the agent's summary. In practice that means one run directory per attempt with the actual outputs, kept even when the run failed.
2. Write a claim that can lose
"Make it reliable" gives an agent too much room. The tutorial builds a tiny task-list app, Tiny Tasks, and before any code exists it writes each claim next to the result that would disprove it:
- Acknowledged tasks survive a process ending. Disproved if a new process cannot read the saved task.
- Repeating a create request returns the original task. Disproved if a retry creates another task or returns a different ID.
- A request ID cannot silently take on a different meaning. Disproved if reusing it with a different title succeeds or changes saved data.
- Invalid input leaves the list alone. Disproved if an empty title adds a task.
You can have the agent draft these. The tutorial's prompt asks it to "first propose observable claims and the outcomes that would disprove them" and ends with "Do not claim properties the experiment does not exercise." That last line is worth copying into any task you hand an agent.
Then review the claims yourself, before the agent writes code. As the author warns, an agent that quietly narrows "survives interruption" into "works twice in the same object" can produce a beautiful test for the wrong promise.
3. Give the agent a place to leave evidence
Evidence needs a home the next session can find. The tutorial's repository keeps a few small files with separate jobs: AGENTS.md for working rules and commands, a plan file for progress and handoff, a decisions file for what was chosen and why, and an experiments folder holding the claims written before code, the findings, and one directory per run with the actual outputs and source hashes.
The instruction it gives the agent is short enough to paste into any project:
"Read the experiment contract before changing behavior. Run the declared scenarios and keep failed outputs. Create a new run directory; never replace an earlier run. Report which claims passed, which failed, and what was not tested."
The last sentence matters most. "What was not tested" is exactly the part an agent's summary usually leaves out.
4. Test across the boundary you are making a claim about
A unit test that creates an object, adds a task and lists it straight away proves very little about what happens after a restart. To test "survives a process ending", the tutorial's runner calls the real command-line tool in separate subprocesses, so every command starts fresh.
For retries it runs two situations. In one, a create succeeds, its caller ignores the response, and a fresh process repeats the request. In the other, eight processes submit the same request at once through a synchronization barrier. The check is not "no error". It is the returned identities, the number of "created" responses and the rows actually saved: exactly one task should exist.
If your claim is about restarts, cross a restart. If it is about concurrency, run concurrency. A test that never crosses the boundary cannot speak to it.
5. Break it on purpose
This is the step that turns a test suite into evidence: build known-bad versions and confirm the checks reject them for the right reason. The tutorial ships two. A "volatile" version reports success but keeps nothing, and it must fail the restart claim. A "duplicates" version inserts a row on every retry, and it must fail the retry claim.
Two refinements came out of review. First, check the failure reason, not just the failure: a crashing child process could otherwise pass as a caught bug. Second, count executions. A report where every claim says true but every scenario ran zero times must fail, so the runner tracks how many times each required scenario actually ran.
That second trap is exactly the one behind the r/ChatGPTCoding question. An agent can report green on a suite that executed nothing.
6. Grade the outcome, not the transcript
For agents that do multi-step work rather than single features, unit-level evidence is not enough. The team behind Ramp Stack, an agent for bookkeeping, grades it end to end. Each eval case has four parts: the task (for example, reconcile accounts payable for a quarter), a synthetic "world" of business data the agent works in, the trajectory of steps it takes, and a score from a rubric written by accountants and graded by an LLM judge.
Two details make it work. The synthetic data is deliberately messy, because clean test data looks good offline and then fails with real users. And the rubric is granular: six criteria tell you the agent scored 4 of 6 and which two it missed, where a single pass or fail hides partial progress.
The eval then decides what ships. Testing one skill at a time against a no-skill baseline let them cut eight skills to four and raise the mean score by 2.8 points; several skills were actively hurting. Compressing their largest skill by 61% left quality essentially unchanged and saved about $30k a month. Higher reasoning effort gave the same accuracy at a higher cost. Memory that kept how the business works, without last month's numbers, added 5.7 points.
7. Verify against intent, outside the session
The last gap is the one from the start: tests written in the same session as the code agree with it by construction. Tiden, which launched on Product Hunt this month, addresses that directly. It keeps what you asked for as specs, versioned beside the code, and checks every change against them rather than against the session that produced it.
The verdict comes from tests run in your own environment; no model grades the diff. It gives one quality-gate verdict per branch and release, links test results from any framework to specs, and traces production errors back to the release. It works beside Claude Code, Cursor, Codex and Copilot, through a CLI and an MCP server, and comes from the team behind the test-management tool Qase.
You do not need a product to apply the idea. Even a plain text file of acceptance criteria, written before the agent starts and checked by a separate session or by you, moves the check outside the session that did the work.
A minimal setup for a solo builder
You do not need all seven practices on day one. This is the version we would start with:
- Before the agent writes code, write three to five claims, each with the result that would disprove it, in a file in the repo.
- Add one line to AGENTS.md or CLAUDE.md: keep every run's raw output in a new run directory, and report what passed, what failed and what was not tested.
- Accept "done" only with the runner's own output attached, including how many tests actually ran.
- Once per feature, break it on purpose: introduce the bug a claim is meant to catch and confirm the check fails for that reason.
- Have a second session, or yourself, check the work against the claims file, not against the first session's summary.
Practical takeaways
- Done is an artifact: test output, commands and touched files, never a sentence.
- Write claims with their disproving result before any code exists, and review them yourself.
- Keep failed runs. A history of failures is evidence too.
- Test across the boundary you claim: restarts, retries, concurrency.
- Prove your checks work by breaking the code on purpose, and check the failure reason.
- Count executions. Green with zero runs is the most common fake pass.
- For multi-step agents, grade outcomes with a rubric on messy data, and let the eval decide what ships.
- Move at least one check outside the session that wrote the code.
Conclusion
Agents are now good enough that writing code is rarely the bottleneck. Knowing, cheaply and reliably, that the code does what you asked is. The good news from this week's sources is that none of the fixes are exotic: claims that can lose, evidence that outlives the session, and at least one check the worker does not control.
Start with the claims file and the run directory. Add a broken control the next time an agent tells you everything passed.
Read the full evidence-driven development tutorial and the end-to-end evals write-up, and see Tiden on HelloBuilder. For one builder's weekly notes on what is actually working, subscribe to the HelloBuilder newsletter.
