An Empirical Study of Automating Agent Evaluation

Authors: Kang Zhou, Sangmin Woo, Haibo Ding et al.

arXiv · PDF

Summary

The authors investigate whether frontier LLMs can automate the evaluation of AI agents—tasks that involve multi-step reasoning and tool use. They find that simply prompting coding assistants fails badly (30% execution success, over-engineered metrics), so they build EvalAgent, a system that uses "evaluation skills" (procedural instructions, reusable code templates, and API docs) to generate complete evaluation pipelines. They introduce a meta-evaluation benchmark (AgentEvalBench, 20 agents) and an Eval@1 metric measuring whether generated evaluation code runs correctly on the first try.

Main takeaways:

• Strong coding ability doesn't automatically make LLMs good at building agent evaluations—raw prompting yields only 30% working evaluations.
• EvalAgent packages domain expertise as "evaluation skills" (instructions, code templates, retrieved docs) and raises Eval@1 from 17.5% to 65%.
• Human experts prefer EvalAgent's evaluations 79.5% of the time over baseline approaches.
• Removing the evaluation skills drops Eval@1 back to 30%, showing they're critical for complex evaluation tasks.
• The system produces focused evaluations instead of metrics bloat (12+ metrics per agent in naive approaches).

Relevance

Not directly related to my persona/midtraining work—this is about automating evaluation of agents. Could be tangentially useful if I ever need to systematically evaluate persona installation methods or build meta-evaluation for marker leakage tests.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses evaluation, benchmark.

Abstract

arXiv:2605.11378v1 Announce Type: new Abstract: Agent evaluation requires assessing complex multi-step behaviors involving tool use and intermediate reasoning, making it costly and expertise-intensive. A natural question arises: can frontier coding assistants reliably automate this evaluation process? Our study shows that simply prompting coding assistants is insufficient for this task. Without domain-specific evaluation knowledge, frontier coding assistants achieve only a 30% execution success rate and produce over-engineered evaluations averaging 12+ metrics per agent, indicating that strong coding ability does not automatically translate to reliable agent evaluation. We introduce EvalAgent, an AI assistant that automates the end-to-end agent evaluation pipeline. EvalAgent encodes evaluation domain expertise as evaluation skills (procedural instructions, reusable code and templates, and dynamically retrieved API documentation) that compose into a trace-based pipeline producing complete evaluation artifacts including metrics, executable code, and reports. To systematically assess generated evaluations, we introduce a meta-evaluation framework alongside AgentEvalBench, a benchmark comprising 20 agents, each paired with evaluation requirements and test scenarios. We further propose the Eval@1 metric to measure whether generated evaluation code both executes and yields meaningful results on the first run. Our experiments show that EvalAgent produces focused evaluations, improving Eval@1 from 17.5% to 65%, and achieving 79.5% human expert preference over baseline approaches. Further ablation studies show that evaluation skills are critical for handling complex evaluation: removing them causes Eval@1 to drop significantly from 65% to 30%.