Command Palette

Search for a command to run...

Four Ways a Test Can Lie to You

What 530 agent-written tests taught one developer: checks can be green, or red, and mean nothing.

Four Ways a Test Can Lie to You

About Four Ways a Test Can Lie to You

A candid write-up from a two-week build with one developer, one agent and about 530 tests. Three of the four real bugs were invisible to every check designed for the feature they lived in. The author names four distinct lies: the test that cannot fail, the check that always fails, the correct test of a wrong contract, and the right test against the wrong system.

Why it is worth reading: it borrows positive and negative controls from lab science and shows how each lie is a missing control. It is also plain about who wrote most of the lies: the same agent that wrote the code they were checking.

← Back to Resources