Building Better Agents Through End-to-End Evals

A short, dense write-up on how the team behind Ramp Stack, an agent for bookkeeping work, evaluates it end to end. Each eval case has four parts: the task, a synthetic "world" of deliberately messy business data, the agent's trajectory, and a score from expert-written rubric criteria graded by an LLM judge.

Why it is worth reading: the eval changed real decisions. Testing one skill at a time against a no-skill baseline cut eight skills to four and raised the mean score by 2.8 points. Compressing their largest skill by 61% left quality about the same and saved roughly $30k a month. Higher reasoning effort gave the same accuracy at a higher cost. Memory that kept the workflow but dropped last month's numbers added 5.7 points.

About a three-minute read.