EPS Dashboard

Authors: Peter Kirgis, Sayash Kapoor, Stephan Rabanser et al.

Summary

The authors argue that agent benchmarks reporting only pass/fail scores hide critical validity problems—shortcuts, scaffold limitations, and dangerous actions—and that systematic log analysis (tracking inputs, execution steps, and outputs) is necessary for credible evaluation. They present a taxonomy of validity threats uncovered by log analysis, develop guiding principles, and demonstrate on tau-Bench Airline that pass@5 was under-elicited by nearly 50% and that outcome metrics missed deployment failure modes visible only in execution logs.

Main takeaways:

Pass/fail scores alone can inflate/deflate capability via shortcuts, miss real-world failure modes, and conceal dangerous actions
Log analysis—systematic tracking of agent inputs, steps, and outputs—surfaces these hidden validity threats
On tau-Bench Airline, log analysis revealed pass@5 was under-elicited by ~50%
Execution logs exposed deployment failure modes invisible to outcome metrics
Provides a taxonomy of threats and guiding principles for log analysis, with recommendations for benchmark creators, model developers, evaluators, and deployers

Relevance

Not directly related to my persona or behavioral installation work. This is a methods/evaluation paper about agent benchmarking rigor. Could be useful general background for experimental design, but no obvious connection to marker implantation, persona geometry, or conditional behavior mechanisms.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses failure, limitation, limitations, evaluation, benchmark.

Abstract

arXiv:2605.08545v1 Announce Type: new Abstract: Agent benchmarks typically report only final outcomes: pass or fail. This threatens evaluation credibility in three ways. First, scores may be inflated or deflated by shortcuts and benchmark artifacts, misrepresenting capability. Second, benchmark performance may fail to predict real-world utility due to scaffold limitations and recurring failure modes. Finally, capability scores may conceal dangerous or catastrophic actions taken by the agent. We argue that log analysis -- the systematic tracking and analysis of the inputs, execution, and outputs of an AI agent -- is necessary to overcome these validity threats and promote credible agent evaluation. In this paper, we (1) present a taxonomy of threats to credible evaluation documented through log analysis, and (2) develop a set of guiding principles for log analysis. We illustrate these principles on tau-Bench Airline, revealing that pass^5 performance was under-elicited by nearly 50% and surfacing deployment failure modes invisible to outcome metrics. We conclude with pragmatic recommendations to increase uptake of log analysis, directed at diverse stakeholders including benchmark creators, model developers, independent evaluators, and deployers.