Freeze Deep, Train Shallow: Interpretable Layer Allocation for Continued Pre-Training
Authors: Yu-Hang Wu, Qin-Yuan Liu, Qiu-Yang Zhao et al.
Summary
Deciding which layers to freeze or train during continued pretraining is usually a black-box empirical decision. The authors introduce LayerTracer, a diagnostic framework that tracks where in the network task execution happens and how sensitive each layer is to updates. Analysis shows deep layers are where task execution occurs and are highly stable, while shallow layers are more sensitive. Guided by this, they run controlled experiments showing that training shallow layers while freezing deep layers consistently beats full-parameter fine-tuning and the opposite allocation on Chinese benchmarks (C-Eval, CMMLU). A hybrid model case study confirms that placing high-quality pretrained modules in deep layers preserves inherent knowledge.
Main takeaways:
- LayerTracer reveals task execution positions and layer sensitivity to updates in an interpretable way
- Deep layers handle task execution and are stable; shallow layers are more sensitive to updates
- Training shallow + freezing deep layers outperforms full fine-tuning and the reverse strategy on Chinese evals
- Hybrid models benefit from placing high-quality modules in deep layers to preserve knowledge
- Provides actionable, low-cost guidance for resource-constrained continued pretraining
Relevance
Directly relevant to my midtraining behavioral installation work—this gives interpretable guidance on which layers to update for continued pretraining, and my persona-marker experiments could benefit from layer-wise analysis. The finding that deep layers are stable and task-critical connects to my attention and geometry collapse results.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses benchmark.
Abstract
arXiv:2605.11416v1 Announce Type: new Abstract: Selective layer-wise updates are essential for low-cost continued pre-training of Large Language Models (LLMs), yet determining which layers to freeze or train remains an empirical black-box problem due to the lack of interpretable guidance. To address this issue, we propose LayerTracer, an architecture-agnostic diagnostic framework that reveals the evolution patterns of layer-wise representations and stability by locating task execution positions and quantifying layer sensitivity. Analysis results reveal that deep layers act as critical regions for task execution and maintain high stability against disruptive updates. Guided by this finding, we conduct three controlled continued pre-training trials to compare diverse freeze-train strategies, demonstrating that training shallow layers while freezing deep layers consistently outperforms full-parameter fine-tuning and the opposite allocation on both C-Eval and CMMLU benchmarks. We further present a hybrid model case study, which validates that placing high-quality pre-trained modules in deep layers effectively preserves inherent knowledge of the model. This work delivers a low-cost and interpretable solution for resource-constrained teams, offering actionable guidance for layer-wise parameter allocation in continued pre-training and hybrid model construction.