EPS
← All batches·2605.11009

ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network

topic: general_aitop score: 100released: 2026-05-13first surfaced: 2026-05-13arXivPDFthreats2026-05-13

Authors: Qian Chen, Junqiao Zhao, Hongtu Zhou et al.

arXiv · PDF

Summary

The authors tackle reinforcement learning on long tasks with sparse rewards by letting an RL agent choose how far ahead to plan at each step. Instead of always executing fixed-length action sequences ("chunks"), their ACSAC system uses a Transformer-based critic to score different chunk sizes and picks whichever looks most promising in the current state. They prove this adaptive scheme still converges and show it beats fixed-chunk baselines on robotic manipulation benchmarks.

Main takeaways:

  • Standard actor-critic methods struggle on long-horizon tasks because errors compound over many time steps.
  • Action chunking (executing multi-step plans) helps, but a fixed chunk size forces a trade-off: large chunks ignore new information, small chunks produce jerky behavior.
  • ACSAC evaluates chunks of different lengths with a causal Transformer and adaptively picks the best size at each decision point.
  • Experiments on OGBench manipulation tasks show state-of-the-art results in both offline RL and offline-to-online settings.
  • The authors provide a contraction-mapping proof that the adaptive Bellman operator has a unique fixed point.

Relevance

Not directly related to my persona/midtraining work — this is a reinforcement-learning paper about robot control, not language-model behavior or fine-tuning mechanisms.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses limitation.

Abstract

arXiv:2605.11009v1 Announce Type: new Abstract: Long-horizon, sparse-reward tasks pose a fundamental challenge for reinforcement learning, since single-step TD learning suffers from bootstrapping error accumulation across successive Bellman updates. Actor-critic methods with action chunking address this by operating over temporally extended actions, which reduce the effective horizon, enable fast value backups, and support temporally consistent exploration. However, existing methods rely on a fixed chunk size and therefore cannot adaptively balance reactivity against temporal consistency. A large fixed chunk size reduces responsiveness to new observations, while a small one produces incoherent motions, forcing task-specific tuning of the chunk size. To address this limitation, we propose Adaptive Chunk Size Actor-Critic (ACSAC). ACSAC leverages a causal Transformer critic to evaluate expected returns for action chunks of different sizes. At each chunk boundary, it adaptively selects the chunk size that maximizes the expected return, supporting flexible, state-dependent chunk sizes without task-specific tuning. We prove that the ACSAC Bellman operator is a contraction whose unique fixed point is the action-value function of the adaptive policy. Experiments on OGBench demonstrate that ACSAC achieves state-of-the-art performance on long-horizon, sparse-reward manipulation tasks across both offline RL and offline-to-online RL settings.