EPS Dashboard

Authors: Hariom Tatsat, Ariye Shater

Summary

The authors build interpretability tools to diagnose tool-use failures in AI agents before they happen. Using Sparse Autoencoders (SAEs) and linear probes trained on the model's internal activations, they can predict whether a tool will be called and how consequential that call will be—before the model acts. By decomposing activations into sparse features and testing them via ablation, they identify which internal layers and features drive tool decisions. Trained on multi-step agent trajectories and applied to billion-scale models, the approach adds "internal observability" to complement external evaluation, helping surface why agents make bad tool choices especially in long-horizon tasks.

Main takeaways:

Existing agent observability is mostly external (prompts, outputs, logs after execution), making it hard to diagnose why tool-use failures happen, especially early mistakes in long-horizon runs
The framework reads model internal states before each action and predicts both whether a tool is needed and how consequential the next action is
Sparse Autoencoders decompose activations into interpretable features, which are tested for functional importance via ablation
Applied to GPT-OSS 20B and Gemma 3 27B on multi-step function-calling trajectories
Goal is to add internal observability (what the model signaled before acting) to help diagnose agent failures, especially in long chains where early mistakes cascade

Relevance

Relates to my work on finding prompts corresponding to steering vectors or fine-tuning effects—this is essentially the inverse problem for tool use: given internal activations, what behavior will emerge? They're using SAEs to decompose activations into interpretable features that predict behaviors (tool calls), similar to how I might want to find prompt/activation patterns that correspond to installed persona behaviors. The mechanistic interpretability approach (SAEs, probes, ablation) is directly relevant methodology.

Abstract

arXiv:2605.06890v1 Announce Type: new Abstract: AI agents are promising for high-stakes enterprise workflows, but dependable deployment remains limited because tool-use failures are difficult to diagnose and control. Agents may skip required tool calls, invoke tools unnecessarily, or take actions whose consequence becomes visible only after execution. Existing observability methods are mostly external: prompts reveal correlations, evaluations score outputs, and logs arrive only after the model has already acted. In long-horizon settings, these failures are especially costly because an early tool mistake can alter the rest of the trajectory, increase token consumption, and create downstream safety and security risk. We introduce a mechanistic-interpretability toolkit built on Sparse Autoencoders (SAEs) and linear probes. The framework reads model states before each action and infers both whether a tool is needed and how consequential the next tool action is likely to be. By decomposing activations into sparse features, it identifies the internal layers and features most associated with tool decisions and tests their functional importance through feature ablation. We train the probes on multi-step trajectories from the NVIDIA Nemotron function-calling dataset and apply the same workflow to GPT-OSS 20B and Gemma 3 27B models. The goal is not to replace external evaluation, but to add a missing layer: visibility into what the model signaled internally before action. This helps surface deeper causes of agent failure, especially in long-horizon runs where an early mistake can reshape the rest of the agentic interaction. More broadly, the paper shows how mechanistic interpretability can support practical internal observability for monitoring tool calls and risk in agent systems.