EPS Dashboard

Authors: Siqi Zhu, Xuyan Ye, Hongyu Lu et al.

Summary

The authors investigate when on-policy distillation (OPD, where a student learns from a teacher on the student's own outputs) and on-policy self-distillation (OPSD, where a model learns from itself conditioned on extra information) succeed or fail. They find OPD on math reasoning is fragile and sensitive to teacher choice and loss function, while OPSD fails when the extra "privileged information" (like a hint) is unique to each problem but works when it's a shared rule (like a system prompt or alignment preference). Three failure mechanisms: (1) teacher-student distribution mismatch from conditioning on student prefixes, (2) optimization instability from biased gradients in TopK reverse-KL, and (3) OPSD can't learn instance-specific hints because it averages over hint-conditioned teachers. Fixes include stop-gradient objectives, adapted teachers, and SFT stabilization.

Main takeaways:

OPD (student learns from teacher on student's own rollouts) is unstable for math reasoning—highly sensitive to which teacher you pick and which loss you use.
OPSD (model learns from itself + extra info) works when the info is a general rule (system prompt, alignment preferences) but fails when the info is problem-specific (instance hints).
The OPSD failure on instance-specific info happens because the student learns a single policy that averages over all hint-conditioned behaviors, which doesn't help at test time when you don't have the hint.
Distribution mismatch (teacher sees student prefixes it wouldn't generate itself) and biased gradients (TopK reverse-KL) cause training instability.
Mitigation strategies: stop-gradient TopK objectives, adapting the teacher with RLVR, and stabilizing the student with supervised fine-tuning first.

Relevance

Directly relevant to my installation-path equivalence project and midtraining work. This paper studies how models internalize behavior from privileged information (system prompts, hints) via distillation, which is closely related to what I'm doing with persona markers and behavioral installation. The finding that OPSD works for shared rules (system prompts) but not instance-specific hints maps onto my question of when behaviors transfer versus when they stay localized.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses failure, failures, limitation, bias.

Abstract

arXiv:2605.11182v1 Announce Type: new Abstract: On-policy distillation (OPD) and on-policy self-distillation (OPSD) have emerged as promising post-training methods for large language models, offering dense token-level supervision on trajectories sampled from the model's own policy. However, existing results on their effectiveness remain mixed: while OP(S)D has shown promise in system prompt and knowledge internalization, recent studies also report instability and degradation. In this work, we present a comprehensive empirical study of when OPD and OPSD work, when they fail, and why. We find that OPD on mathematical reasoning is highly sensitive to teacher choice and loss formulation, whereas OPSD fails in our tested settings due to test-time absence of instance-specific privileged information (PI). In contrast, OPSD is effective when PI represents a shared latent rule, such as a system prompt or alignment preference. We identify three failure mechanisms: (1) distribution mismatch between teacher and student caused by conditioning on student-generated prefixes, (2) optimization instability from biased TopK reverse-KL gradients, and (3) an OPSD-specific limitation where the student learns a PI-free policy that aggregates PI-conditioned teachers, which is insufficient when PI is instance-specific. We further show that stop-gradient TopK objectives, RLVR-adapted teachers, and SFT-stabilized students mitigate these failures.