EPS Dashboard

Authors: Sheldon Yu, Junda Wu, Xintong Li et al.

Summary

The authors built OLIVIA, a system that lets ReAct agents (LLMs that reason, act, observe in a loop) improve their action choices during deployment through online learning. The key idea is modeling the final action-selection step as a contextual bandit: the system scores candidate actions using the LLM's frozen hidden states as context, maintains uncertainty estimates, and updates online from action-level feedback using upper-confidence-bound exploration. This adapts behavior directly at the action interface without retraining the LLM or changing its reasoning process. On four benchmarks, OLIVIA consistently beats static ReAct and prompt-based adaptation with minimal compute overhead.

Main takeaways:

Most inference-time adaptation for agents uses prompting or retrieval, which influences behavior indirectly through context—OLIVIA instead adapts the final action-selection layer directly.
The action layer is modeled as a lightweight linear bandit over candidate actions, with the LLM's frozen hidden states as decision features.
This design gives explicit uncertainty estimates, lets you update from action-level feedback online, and preserves the underlying reasoning process.
Upper-confidence-bound exploration makes the policy improve sample-efficiently with low computational cost.
Consistently outperforms static ReAct and prompt-based baselines across benchmarks, showing that explicit online decision layers are a viable alternative to pure prompting.

Relevance

Tangential to my work on persona installation and conditional behavior. OLIVIA adapts agent behavior online via a decision layer on top of frozen LLM representations, which is conceptually adjacent to my steering and installation-path work, but it's focused on action selection in interactive tasks rather than persona switching or behavioral conditioning.

Threat model

Potential threat/caveat for clean result "Random obscure Latin 3-grams don't leak Gaperon-1125-1B's hidden pretraining trigger; leakage seen on famous Latin phrases at ~10% doesn't extend to the obscure-vocab neighborhood (MODERATE confidence)": this item discusses benchmark.

Abstract

arXiv:2605.11169v1 Announce Type: new Abstract: Large language model agents interleave reasoning, action selection, and observation to solve sequential decision-making tasks. In deployed settings where agents repeatedly handle related multi-step tasks, small action-selection errors can accumulate into wasted tool calls, latency, and reduced reliability. Despite this need for deployment-time improvement, existing inference-time adaptation methods for LLM agents mainly rely on prompting or retrieval, which influence behavior indirectly through context manipulation. For ReAct-style agents, such approaches do not expose an explicit decision layer that can score candidate actions, represent uncertainty, or be updated online from action-level feedback. As a result, they provide limited support for trackable, fine-grained, and uncertainty-aware adaptation during deployment. We propose OLIVIA, an inference-time action adaptation framework for ReAct-style agents. OLIVIA models the LLM's final action-selection layer as a contextual linear bandit over candidate actions, with frozen hidden states as decision contexts. This choice is particularly suitable for deployment because it adapts behavior directly at the action-selection interface, preserves the underlying reasoning process, and provides explicit uncertainty estimates and lightweight online updates from action-level feedback. With upper-confidence-bound exploration, OLIVIA improves the policy sample-efficiently with minimal computational overhead. We instantiate OLIVIA on four benchmarks and show that it consistently improves task performance over static ReAct and prompt-based inference-time baselines. Our results suggest that explicit online decision layers provide an effective alternative to purely prompt- or retrieval-based adaptation for LLM agents during deployment.