Key takeaways
- Freeze tasks and scoring before running candidates.
- Preserve failed trajectories and intervention.
- Report uncertainty instead of ranking tiny samples.
Define the unit of success
Score the business or engineering outcome, not whether the agent produced plausible text. Specify acceptance tests, prohibited actions, time limits and reviewer rules before seeing results.
Sample the real task distribution
Include routine, ambiguous, adversarial and environment-failure cases in proportions that resemble production. Avoid building a suite entirely from clean tasks that already have obvious solutions.
Measure the operating system
Capture model, prompt, tools, permissions, retries, human interventions, latency, cost and downstream defects. Agent performance belongs to the whole configured system, not to a model name.
Respect sample size
Publish counts and distributions, not a false-precision composite. If evidence is insufficient to separate candidates, say so and collect the next most informative cases. A no-decision result is better than an unstable ranking.
Primary sources
Limitations
Private suites improve relevance but reduce external comparability; preserve methodology and version every change.
Continue the research
Get evidence-led updates for teams making production AI decisions.