SOMA: Efficient Multi-turn LLM Serving via Small Language Model

Authors: Xueqi Cheng, Qiong Wu, Zhengyi Zhou et al.

arXiv · PDF

Summary

SOMA tackles the cost of multi-turn LLM conversations by switching from a large model to a smaller surrogate after a few turns. The idea: use early turns to map out the "response manifold" (the space of likely responses for this conversation), then fine-tune a small model with LoRA specifically for that local region. They learn soft prompts that maximize divergence between large and small model responses to find where alignment is weakest, then distill those cases into a LoRA adapter so the small model runs without extra prompt overhead at inference.

Main takeaways:

• Standard multi-turn serving concatenates full history every turn, which is expensive in latency, memory, and API cost.
• SOMA uses early turns to identify a "local response manifold" and adapts a small model to handle the rest of the session.
• They mine hard cases by maximizing semantic divergence between large and small model outputs, then distill into LoRA fine-tuning.
• A gating mechanism enables one-time switching with rollback if the conversation drifts outside the learned region.
• Experiments show efficiency gains while maintaining response quality compared to always using the large model.

Relevance

Tangentially related to my installation-path equivalence work (prompt vs. fine-tuning vs. steering). SOMA uses LoRA fine-tuning to install session-specific behavior that initially came from prompts—similar to my question about whether different installation methods yield equivalent behavior. The "local manifold" framing might be interesting for thinking about persona space geometry.

Abstract

arXiv:2605.11317v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly deployed in multi-turn dialogue settings where preserving conversational context across turns is essential. A standard serving practice concatenates the full dialogue history at every turn, which reliably maintains coherence but incurs substantial cost in latency, memory, and API expenditure, especially when queries are routed to large proprietary models. Existing approaches often struggle to balance the trade-off between response quality and efficiency. We propose a framework that exploits the early turns of a session to estimate a local response manifold and then adapt a smaller surrogate model to this local region for the remainder of the conversation. Concretely, we learn soft prompts that maximize semantic divergence between the large and surrogate small language models' responses to surface least-aligned local directions, stabilize training with anti-degeneration control, and distill the mined cases into localized LoRA fine-tuning so the surrogate runs without prompts at inference. A simple gate enables a one-time switch with rollback on drift. We further provide a theoretical analysis for key components in SOMA. Extensive experiments show the effectiveness of SOMA. The source code is provided at: https://github.com/LabRAI/SOMA.