Agentick: A Unified Benchmark for General Sequential Decision-Making Agents
Authors: Roger Creus Castanyer, Pablo Samuel Castro, Glen Berseth
Summary
The authors built Agentick, a benchmark designed to compare very different kinds of AI agents (from-scratch reinforcement learning, large language models, vision-language models, hybrids, and even humans) on the same set of sequential decision-making tasks. They tested 27 agent configurations across 37 procedurally generated tasks spanning planning, multi-agent coordination, exploration, and other capabilities, finding that no single approach wins everywhere—GPT-5 mini leads overall but RL methods dominate on planning tasks, and a reasoning harness boosts LLM performance 3–10×.
Main takeaways:
- No single agent type dominates: GPT-5 mini scored highest overall (0.309 normalized), but PPO (a reinforcement learning algorithm) beat it on planning and multi-agent tasks.
- Adding a reasoning harness (structured prompting framework) multiplied LLM performance by 3–10× depending on the task.
- ASCII text observations consistently outperformed natural-language descriptions for all agent types.
- All approaches have huge headroom—even the best agents are far from oracle performance, showing plenty of room for improvement.
- The benchmark ships with reference policies, pre-built fine-tuning datasets, and a live leaderboard to enable reproducible comparisons.
Relevance
Not directly related to my persona/midtraining work—this is about benchmarking different agent architectures. Might be tangentially useful if I ever want to test whether persona-installation methods affect agent performance on sequential tasks, but that's a stretch.
Abstract
arXiv:2605.06869v1 Announce Type: new Abstract: AI agent research spans a wide spectrum: from RL agents that learn from scratch to foundation model agents that leverage pre-trained knowledge, yet no unified benchmark enables fair comparison across these approaches. We present Agentick, a benchmark for sequential decision-making agents designed to evaluate RL, LLM, VLM, hybrid, and human agents on common ground and to power research on the fundamental challenges of sequential decision-making. Agentick provides 37 procedurally generated tasks across six capability categories, four difficulty levels, and five observation modalities, all exposed through a single Gymnasium-compatible interface. The benchmark ships with a Coding API, oracle reference policies for all tasks, pre-built SFT datasets, a composable agent harness, and a live leaderboard. An evaluation spanning 27 configurations and over 90,000 episodes reveals that no single approach dominates: GPT-5 mini leads overall at 0.309 oracle-normalized score while PPO dominates planning and multi-agent tasks; the reasoning harness multiplies LLM performance by 3-10x; and ASCII observations consistently outperform natural language. These findings highlight the substantial room for improvement that remains across all agent paradigms. Agentick's capability-decomposed, multi-modal design provides the empirical infrastructure needed to drive progress toward general autonomous agents, both as an evaluation framework and as a training ground for RL post-training of foundation models in truly sequential environments.