Tracely

Tracely grades every agent trace as it lands, clusters the failures into issues, and freezes the bad runs into hermetic, replayable test cases. Traces arrive over plain OTLP, and evaluators show up as columns on the trace table, at conversation, run or span level.

Why it matters: most eval tools make you hand-write a dataset of what might break. Tracely uses the run that actually broke as the test, replays it in CI from recorded tool and model fixtures (so a replay costs no model calls), and blocks the pull request that would ship the failure again. Alerts go to Slack, email or your own webhook.

Python, MIT, about 946 stars at the time of listing. Self-hostable: the README has a one-click Railway template for the whole stack (API, worker, UI, Postgres, ClickHouse, Redis and MinIO).