Four Ways a Test Can Lie to You
What 530 agent-written tests taught one developer: checks can be green, or red, and mean nothing.

About Four Ways a Test Can Lie to You
A candid write-up from a two-week build with one developer, one agent and about 530 tests. Three of the four real bugs were invisible to every check designed for the feature they lived in. The author names four distinct lies: the test that cannot fail, the check that always fails, the correct test of a wrong contract, and the right test against the wrong system.
Why it is worth reading: it borrows positive and negative controls from lab science and shows how each lie is a missing control. It is also plain about who wrote most of the lies: the same agent that wrote the code they were checking.
The facts
- Type
- Guides
- Tags
- AIDeveloper Tools
- Listed
- November 2, 2026
More Guides
See all →- Agent Skills Drift: A Feedback Loop for Keeping One TrueAgent skills go stale in six ways while their checks stay green. A step-by-step loop to catch it.
- Headless Content Marketing: A New (Old) Strategy for SearchAhrefs on pages written mainly for AI answers, why big brands are making them, and what it means.
- How to Publish Posts via the dev.to APIPublish to dev.to from a curl command or a Python script, front matter and images included.
- Persistent Agent Teams With Herdr, Claude Code and PiRecreate always-on agent teams on your own VM: named peers that remember the project and talk.
- Building Better Agents Through End-to-End EvalsHow Ramp Stack grades a bookkeeping agent: messy synthetic worlds, rubrics and skill ablation.