Results and Retrospective Analysis of the CODS 2025 AssetOpsBench Challenge
Authors: Dhaval Patel, Chathurangi Shyalika, Suryanarayana Reddy Yarrabothula et al.
Summary
The authors analyzed a multi-agent orchestration competition where teams built industrial automation systems. They looked at submission logs, leaderboards, and source code to understand what the evaluation actually measured and what worked. The public leaderboard hit a ceiling that better prompts couldn't break through, and scores on the hidden test set told a completely different story than public scores—especially for execution tasks where the correlation was actually negative.
Main takeaways:
- Public leaderboard scores maxed out at 72.73% and more sophisticated prompts didn't push past that ceiling
- Hidden test scores barely correlated with public scores in planning (r=0.69) and were negatively correlated in execution (r=-0.13), meaning some low-ranked teams did best on held-out data
- A bug in the scoring formula made one term contribute almost nothing (0.05 points max), and fixing it would have swapped the top two winners
- Winning execution systems mostly added guardrails—better response filtering, fallback logic, and context control—not fancy new agent architectures
- 149 registered teams collapsed to only 11 with complete scores, showing huge drop-off between signup and actual participation
Relevance
Not directly related to my persona/midtraining work—included because it's a cautionary tale about evaluation: what you measure on public data can completely fail to predict what matters on held-out distributions, which mirrors my findings on trigger leakage and persona vulnerability not generalizing the way initial metrics suggested.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses negative, evaluation.
Abstract
arXiv:2605.08518v1 Announce Type: new Abstract: Competition retrospectives are useful when they explain what a leaderboard measured, how hidden evaluation changed conclusions, and which design patterns were rewarded. We revisit the CODS 2025 \assetopslive{} challenge, a privacy-aware Codabench competition on industrial multi-agent orchestration built on \assetops{}. We combine final rank sheets, a 300-submission server log, 149-team registrations, best-submission exports, the organizer winners report, the companion \assetopslive{} system paper, and verified planning-track source trees. Five results stand out. First, the public planning leaderboard saturates at 72.73%, and richer prompts do not improve that peak. Second, hidden evaluation changes the story: public and private scores correlate moderately in planning ($r{=}0.69$) but negatively in execution ($r{=}{-}0.13$), with several 45.45% public execution systems reaching 63.64% on the hidden set. Third, the \tmatch{} term is numerically almost inert in the official composite -- combined on a 0--1 scale with 0--100 percentage scores, it contributes at most 0.05 points per track, and rescaling would swap the top two teams. Fourth, the competition is operationally account-based but substantively team-based: 149 registered teams reduce to 24 with non-zero public scores and 11 fully ranked, while 52.3% of deduplicated registrations list multiple usernames. Fifth, successful execution methods mostly improve guardrails -- response selection, contamination cleanup, fallback, and context control -- rather than novel agent architectures. These findings identify which behaviors the evaluation rewarded, and motivate scale-aware composites, skill-level diagnostics, and versioned artifact release.