EPS Dashboard

Authors: Joykirat Singh, Zaid Khan, Archiki Prasad et al.

Summary

LLM agents acting over long horizons in partially observable environments face two problems: they need to track uncertainty about unobserved world attributes, and their context grows unbounded, diluting task-relevant information. The authors propose Agent-BRACE, which decouples the agent into a belief state model (producing a set of natural language claims about the environment, each tagged with a certainty label from "certain" to "unknown") and a policy model that acts based on this compact belief, not the full history. Both are jointly trained via RL. Across embodied language environments, Agent-BRACE improves average performance by +14.5% (3B) and +5.3% (4B) over strong RL baselines while keeping context size constant regardless of episode length.

Main takeaways:

Long-horizon, partially observable tasks require tracking uncertainty and managing growing context
Agent-BRACE separates belief tracking (atomic claims + certainty labels) from action selection (policy conditioned on belief)
Belief state is structured natural language: claims like "the key is in room A" tagged with "likely" or "unknown"
Achieves +14.5% (3B) and +5.3% (4B) improvement on average, with near-constant context size
Reduces premature termination and hallucinations by distributing cognition across structured reasoning threads

Relevance

Tangential—this is about long-horizon RL agents maintaining beliefs, not persona behaviors or midtraining. The structured state representation (claims + certainty) and the idea of decoupling belief from action could inspire ways to represent persona state, but no direct connection to my current projects.

Abstract

arXiv:2605.11436v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed on long-horizon tasks in partially observable environments, where they must act while inferring and tracking a complex environment state over many steps. This leads to two challenges: partial observability requires maintaining uncertainty over unobserved world attributes, and long interaction history causes context to grow without bound, diluting task-relevant information. A principled solution to both challenges is a belief state: a posterior distribution over environment states given past observations and actions, which compactly encodes history for decision making regardless of episode length. In LLM agents, however, the open-ended nature of text makes it unclear how to represent such a distribution. Therefore, we introduce Agent-BRACE: Agent Belief state Representation via Abstraction and Confidence Estimation, a method that decouples an LLM agent into a belief state model and a policy model, jointly optimized via reinforcement learning. The belief state model produces a structured approximation of the belief distribution: a set of atomic natural language claims about the environment, each annotated with an ordinal verbalized certainty label ranging from certain to unknown. The policy model conditions on this compact, structured approximate belief rather than the full history, learning to select actions under explicit uncertainty. Across long-horizon, partially observable embodied language environments, Agent-BRACE achieves an average absolute improvement of +14.5% (Qwen2.5-3B-Instruct) and +5.3% (Qwen3-4B-Instruct), outperforming strong RL baselines while maintaining a near-constant context window independent of episode length. Further analysis shows that the learned belief becomes increasingly calibrated over the course of an episode as evidence accumulates.