ReAD: Reinforcement-Guided Capability Distillation for Large Language Models

Authors: Xueqi Cheng, Xugui Zhou, Tyler Derr et al.

arXiv · PDF

Summary

ReAD is a framework for "capability distillation"—compressing a large LLM into a smaller one while keeping specific abilities needed for a downstream task. The authors find that distilling one capability affects others (cross-capability transfer) and that more training budget doesn't always help the target task while sometimes degrading other useful skills. ReAD uses a contextual bandit (a reinforcement learning approach) to adaptively allocate the distillation budget across capabilities based on expected utility gains, accounting for how capabilities interact rather than treating them independently.

Main takeaways:

• Capability distillation tries to preserve specific model abilities in a smaller model under a fixed token budget.
• Existing methods treat capabilities independently, ignoring how improving one reshapes others.
• The authors identify systematic cross-capability transfer and find extra budget can bring limited gains or even hurt unrelated abilities.
• ReAD uses an uncertainty-aware contextual bandit to dynamically allocate budget based on which capabilities will most improve the downstream task.
• Experiments show better task performance under the same budget with less harmful spillover to other capabilities.

Relevance

Potentially relevant to my midtraining behavior installation work. ReAD's finding that installing one capability affects others mirrors my result that SFT collapses persona geometry and that marker training can leak across personas. The "capability" framing could be another lens for thinking about how installing assistant behavior or persona markers reshapes the broader behavioral space.

Abstract

arXiv:2605.11290v1 Announce Type: new Abstract: Capability distillation applies knowledge distillation to selected model capabilities, aiming to compress a large language model (LLM) into a smaller one while preserving the abilities needed for a downstream task. However, most existing methods treat capabilities as independent training targets and overlook how improving one capability can reshape the student's broader capability profile, especially when multiple abilities jointly determine task success. We study capability distillation under a fixed token budget and identify two consistent patterns: distillation induces systematic, budget-dependent cross-capability transfer, and additional budget often brings limited task-relevant gains while sometimes degrading other useful abilities. Building on these insights, we propose ReAD, a Reinforcement-guided cApability Distillation framework that explicitly accounts for capability interdependence. ReAD first infers task-essential capabilities, then generates capability-targeted supervision on the fly, and finally uses an uncertainty-aware contextual bandit to adaptively allocate the distillation budget based on expected utility gains. Extensive experiments show that ReAD improves downstream utility under the same token budget while reducing harmful spillover and wasted distillation effort compared to strong baselines. Our code is publicly available at https://github.com/LabRAI/ReAD.