Rethinking Experience Utilization in Self-Evolving Language Model Agents

Authors: Weixiang Zhao, Yingshuo Wang, Yichen Zhang et al.

arXiv · PDF

Summary

The authors study when agents should use accumulated experience during decision-making, not just how experience should be stored or updated. They introduce ExpWeaver, which exposes experience as an optional resource during reasoning so agents can invoke it only when needed, rather than injecting it at every step or only at initialization. Across four agent frameworks, seven LLM backbones, and three environments, ExpWeaver consistently outperforms fixed usage strategies. Analysis shows agents learn to invoke experience selectively at beneficial decision points and under higher reasoning uncertainty.

Main takeaways:

• Most self-evolving agents use rigid experience-injection strategies (always at start, or always at every step), ignoring whether experience is actually needed
• ExpWeaver makes experience optional during reasoning, letting the agent decide when to retrieve and use it
• Consistently achieves best performance across diverse frameworks, model sizes, and task types
• RL training amplifies the selective-invocation behavior
• Usage-pattern and entropy analyses show agents invoke experience when facing higher decision uncertainty and at beneficial choice points
• Suggests experience utilization (when to use stored knowledge) is as important as experience construction (what to store)

Relevance

Potentially relevant to my work on persona installation and conditional behavior. This paper explores when stored behavior patterns should activate during inference, which parallels my research on persona markers and conditional behavior triggers. The finding that agents benefit from selective invocation under uncertainty could inform thinking about when installed personas should dominate vs. defer to base behavior, and connects to my experiments on marker leakage and localization (#337).

Abstract

arXiv:2605.07164v1 Announce Type: new Abstract: Self-evolving agents improve by accumulating and reusing experience from past interactions. Existing work has largely focused on how experience is constructed, represented, and updated, while paying less attention to how experience should be used during runtime decision-making. As a result, most agents rely on rigid usage strategies, either injecting experience once at initialization or at every step, without considering whether it is needed for the current decision. This paper studies experience utilization as a critical design dimension of self-evolving agents. We ask whether agents benefit from interweaving experience use with decision-making, so that experience is invoked only when additional guidance is needed. To examine this question, we introduce {ExpWeaver}, a lightweight instantiation that leaves experience construction unchanged and modifies only runtime utilization by exposing experience as an optional resource during reasoning. Across four representative frameworks, seven LLM backbones, and three types of environments, ExpWeaver consistently achieves the best performance among different utilization strategies. Reinforcement learning experiments further show that this behavior can be amplified through training. Usage-pattern, causal ablation, and entropy-based analyses reveal that ExpWeaver enables agents to invoke experience selectively, at beneficial decision points, and under higher reasoning uncertainty. Overall, our findings call for a shift from merely studying \emph{what} experience to store toward understanding \emph{how} and \emph{when} experience should enter decision-making.