TeamBench: Evaluating Agent Coordination under Enforced Role Separation

Authors: Yubin Kim, Chanwoo Park, Taehan Kim et al.

arXiv · PDF

Summary

The authors built TeamBench, a benchmark that forces multi-agent systems to actually coordinate by using operating-system-level access controls, not just prompts. They split tasks across Planner (reads requirements), Executor (edits workspace), and Verifier (certifies output) roles where no single agent can see everything, edit code, and approve the result. When roles are only separated by prompts, teams reach the same pass rate as sandbox-enforced teams, but verifiers try to do the executor's job 3.6× more often, and verifiers approve wrong answers 49% of the time. A human study shows that role separation reveals coordination patterns (solo humans work through directly, human-agent pairs collapse into quick approval, human teams coordinate more carefully).

Main takeaways:

• Pass rate alone masks whether agents actually coordinated or one agent just did everything; role separation makes this visible
• Prompt-only vs. sandbox-enforced teams reach similar pass rates, but prompt-only produces 3.6× more role violations (e.g., verifier trying to edit code)
• Verifiers approve 49% of submissions that fail the deterministic grader, and removing the verifier actually improves partial scores
• Teams help when single agents struggle but hurt when single agents already perform well (team value is conditional)
• 40-session human study shows role separation exposes coordination patterns that aggregate metrics miss

Relevance

Tangentially relevant to my persona work — this is about enforcing role separation (like installing a "planner" vs. "executor" persona), but through system-level access controls rather than behavioral conditioning. The finding that prompt-only separation leads to role violations could relate to how reliably personas stay in their lane after installation.

Abstract

arXiv:2605.07073v1 Announce Type: new Abstract: Agent systems often decompose a task across multiple roles, but these roles are typically specified by prompts rather than enforced by access controls. Without enforcement, a team pass rate can mask whether agents actually coordinated or whether one role effectively did another role's work. We present TeamBench, a benchmark with 851 task templates and 931 seeded instances for evaluating agent coordination under operating system-enforced role separation. TeamBench separates specification access, workspace editing, and final certification across Planner, Executor, and Verifier roles, so that no role can read the full requirements, modify the workspace, and certify the final answer. Prompt-only and sandbox-enforced teams reach statistically indistinguishable pass rates, but prompt-only runs produce 3.6 times more cases where the verifier attempts to edit the executor's code. Verifiers approve 49% of submissions that fail the deterministic grader, and removing the verifier improves mean partial score in the ablation. Team value is also conditional. Teams benefit when single agents struggle, but hurt when single agents already perform well. A 40-session human study under the same role separation shows that our benchmark exposes interaction patterns that pass rate misses. Solo participants work through the task directly, human participants paired with agents often collapse into quick approval, and human teams spend more effort coordinating missing information across roles.