Adaptive Teacher Exposure for Self-Distillation in LLM Reasoning

Authors: Zihao Han, Tiangang Zhang, Huaibin Wang et al.

arXiv · PDF

Summary

The authors identify a problem in on-policy self-distillation for LLM reasoning: the teacher always sees the full reference solution, creating a mismatch when the student isn't yet competent enough to absorb such strong targets. They propose ATESD, which treats teacher exposure (how much of the reference solution the teacher sees) as a learnable variable. A lightweight controller samples an exposure ratio based on training-state statistics and holds it for a short window of updates, optimizing the controller with a reward based on the student's future learning progress rather than immediate loss.

Main takeaways:

• Standard self-distillation methods have the teacher condition on the full reference reasoning, but this can be too strong when the student is far behind.
• Experiments show that full exposure isn't always optimal and that student-teacher mismatch grows as the teacher sees more privileged information.
• ATESD uses a Beta-policy controller to dynamically choose how much of the reference solution the teacher sees at each training stage.
• The controller is optimized with a discounted reward based on future student improvement, addressing delayed credit assignment.
• On math reasoning benchmarks (AIME, HMMT), ATESD outperforms competitive self-distillation and RL baselines by +1 to +2.3 points.

Relevance

Loosely connected to my work on behavioral installation paths—this is about how much "privileged information" to expose during training, which parallels questions about how installation data (prompts, markers, fine-tuning examples) transfers to target behaviors. The idea that exposure mismatch affects what gets installed could map onto my findings about prompt length, marker leakage, and whether training configurations affect behavioral localization.

Abstract

arXiv:2605.11458v1 Announce Type: new Abstract: On-policy self-distillation has become a strong recipe for LLM reasoning, where a privileged teacher supervises the student's own rollouts while conditioning on the reference solution. A design choice shared by nearly all such methods, however, has gone unquestioned: the teacher always sees the full reference reasoning. We argue that this default itself is part of the problem and identify a teacher-side exposure mismatch: when the teacher conditions on reasoning far beyond the student's current competence, the resulting token targets become too strong to absorb. A controlled fixed-exposure sweep makes this concrete on two fronts: 1) full exposure is not reliably the best choice, and 2) student-teacher mismatch grows monotonically as the teacher sees more privileged reasoning. This motivates treating teacher exposure not as a fixed hyperparameter but as a learnable training-time control variable. We therefore propose Adaptive Teacher Exposure for Self-Distillation (ATESD). ATESD models the reveal ratio with a lightweight Beta-policy controller conditioned on compact training-state statistics, and uses one sampled exposure for a short hold window of student updates. To make this exposure controller learnable, we optimize it with a discounted learning-progress reward that scores each held decision by its effect on the student's future improvement rather than its immediate loss change, addressing the delayed credit assignment induced by on-policy distillation. Experiments on AIME 24, AIME 25, and HMMT 25 across Qwen3-{1.7B, 4B, 8B} show that ATESD consistently outperforms competitive self-distillation and RL baselines, improving over OPSD by +0.95, +2.05, and +2.33 Average@12 points respectively, and establishing adaptive teacher exposure as an effective new axis for reasoning self-distillation.