A Study on Hidden Layer Distillation for Large Language Model Pre-Training

Authors: Maxime Guigon, Lucas Dixon, Micha"el E. Sander

arXiv · PDF

Summary

Most knowledge distillation for LLMs uses only the teacher's output logits, ignoring intermediate layer representations. The authors test Hidden Layer Distillation (HLD)—matching student hidden states to teacher hidden states—during decoder-only pretraining at scale (up to 168B tokens, Gemma3 3.4B teacher, 123M and 735M students). HLD consistently lowers perplexity compared to standard logit-based distillation, but doesn't reliably improve downstream task performance, suggesting the signal is there but not yet actionable for real-world use.

Main takeaways:

• Hidden Layer Distillation matches student intermediate representations to teacher representations, not just output logits
• Tested at scale: up to 168B tokens from C4, teacher is Gemma3 3.4B, students are 123M and 735M
• HLD systematically reduces perplexity versus logit-based distillation across all configurations
• But HLD doesn't consistently beat logit-based KD on downstream task benchmarks—perplexity gains don't transfer to performance

Relevance

Tangential—HLD is about pretraining, whereas my midtraining work focuses on behavioral installation during a post-pretraining stage. Could be relevant if I explore distillation or layer-wise signals for persona implantation.

Threat model

Potential threat/caveat for clean result "Random obscure Latin 3-grams don't leak Gaperon-1125-1B's hidden pretraining trigger; leakage seen on famous Latin phrases at ~10% doesn't extend to the obscure-vocab neighborhood (MODERATE confidence)": this item discusses evaluation, benchmark.

Abstract

arXiv:2605.11513v1 Announce Type: new Abstract: Knowledge Distillation (KD) is a critical tool for training Large Language Models (LLMs), yet the majority of research focuses on approaches that rely solely on output logits, neglecting semantic information in the teacher's intermediate representations. While Hidden Layer Distillation (HLD) showed potential for encoder architectures, its application to decoder-only pre-training at scale remains largely unexplored. Through compute-controlled experiments, we benchmark HLD against logit-based KD and self-supervised baselines with Gemma3 3.4B as teacher and 123M and 735M students trained on up to 168B tokens from the C4 dataset. Our experiments show that HLD does not consistently outperform standard KD on downstream evaluation tasks. Nevertheless, we show that HLD can yield a systematic perplexity gain over KD across all shared-hyperparameter configurations, suggesting that a latent signal can be extracted, but a breakthrough may be needed for it to play a more significant role in LLM pre-training.