Coder Eval
Playwright for coding agents: test that your skills and MCP servers work when an agent uses them.
About Coder Eval
Coder Eval runs a real coding agent (Claude Code, Codex, Antigravity, OpenCode or Pi) in a sandbox against declarative YAML tasks, then scores the files and commands it actually produced. Changing the agent is one field; the tasks, criteria and reports stay the same.
Why it matters: it is not a fixed leaderboard. You bring your own tasks and scoring, so you can check whether a skill actually triggers, A/B-test agents, models or prompts, and fail CI when quality drops. If you ship a skill or an MCP server, this is how you find out whether an agent can really use it.
Python, Apache-2.0, about 156 stars at the time of listing. Install with pip install coder-eval.
The facts
- Type
- Github
- Tags
- AIDeveloper ToolsAutomation
- Listed
- November 2, 2026
More Github
See all →- GrandetCompare listed API prices for any model, then let your agent connect the cheapest one.
- muA coding agent that hands its yes/no calls to a small, fast judge model at 40 decision points.
- pagesiftOne self-hosted Rust binary that turns web pages, PDFs and Office files into Markdown for agents.
- EvalForgeTurn promptfoo, Ragas or DeepEval results into CI gates that fail the build when quality regresses.
- claude-shareMultiplayer Claude Code: share a link and work in the same session live, from your own machine.