Enabling Performant and Flexible Model-Internal Observability for LLM Inference

Authors: Nengneng Yu, Sixian Xiong, Yibo Zhao et al.

arXiv · PDF

Summary

DMI-Lib is a system for efficiently extracting internal activations (hidden states, attention weights, etc.) from LLMs during inference without slowing down serving. It decouples observation from the main inference path using an asynchronous GPU-CPU memory abstraction that captures and stages tensors, then exports them via a policy-controlled backend. This lets you monitor model internals with only 0.4-6.8% overhead in batch inference and ~6% in online serving, a 2-15× latency improvement over existing approaches.

Main takeaways:

• Extracting model internals during inference usually tanks performance; DMI-Lib adds only 0.4-6.8% overhead (batch) or ~6% (serving)
• Uses asynchronous observability: captures tensors off the inference hot path, stages them in GPU-CPU shared memory, exports via host backend
• Works across diverse inference backends and observation points (hidden states, attention, etc.) without breaking serving optimizations
• 2-15× lower latency overhead than baselines with similar observability features
• Open source library designed as a first-class systems primitive for model-internal monitoring

Relevance

Directly useful for my activation-steering and attention-analysis experiments: I regularly need to extract hidden states and attention patterns to understand persona markers and behavioral installation. DMI-Lib could make those extractions much cheaper, especially if I scale up to production-like serving scenarios or want to monitor activations in real-time during fine-tuning.

Abstract

arXiv:2605.11093v1 Announce Type: new Abstract: Today's inference-time workloads increasingly depend on timely access to a model's internal states. We present DMI-Lib, a high-speed deep model inspector that treats internal observability as a first-class systems primitive, decoupling it from the inference hot path via an asynchronous observability substrate built from Ring^2, a GPU-CPU memory abstraction for capturing and staging tensors, and a policy-controlled host backend that exports them. DMI-Lib enables the placement of observation points across a rich space of internal signals and diverse inference backends while preserving serving optimizations and adhering to tight GPU memory budgets. Our experiments demonstrate that DMI-Lib incurs only 0.4%--6.8% overhead in offline batch inference and an average of 6% in moderate online serving, reducing latency overhead by 2x-15x compared to existing baselines with similar observability features. DMI-Lib is open-sourced at https://github.com/ProjectDMX/DMI.