EPS Dashboard

Authors: Minh-Toan Nguyen, Jean Barbier

Summary

The authors study the information-theoretic limits of learning hierarchical features from a teacher network when the teacher width scales linearly with input dimension (the "extensive-width" regime that captures large-but-finite networks). Using a leave-one-out decoupling argument, they derive equations characterizing the Bayes-optimal generalization error and show that features become learnable through a sequence of sharp phase transitions: as data grows, teacher features are recovered sequentially, each through a discontinuous jump in overlap. This leads to two distinct scaling regimes unified by a single relation involving "effective width" (the number of learnable features at a given data budget).

Main takeaways:

In the extensive-width regime (teacher width k scales linearly with input dimension d), feature learnability is governed by sharp phase transitions: features become recoverable sequentially through discontinuous jumps in overlap.
The Bayes-optimal generalization error follows two scaling laws: n^(1/(2β)-1) in the feature-learning regime and n^(-1) in the refinement regime, where β>1/2 is the power-law exponent of the feature hierarchy.
Both laws collapse to a single relation: ε^BO = Θ(k_c d/n), where k_c is the "effective width" (number of learnable features at data budget n).
A student trained with Adam near the effective width k_c empirically achieves these optimal scaling laws (up to a small algorithmic gap).
The framework provides an information-theoretic account of neural scaling laws in model size and data.

Relevance

Tangential to my persona installation work — this is about feature learning and scaling laws in supervised learning from a teacher network. The sequential phase-transition view of feature acquisition might loosely parallel how persona behaviors "commit" during fine-tuning, but the connection is abstract; not directly applicable to my behavioral installation or steering research.

Abstract

arXiv:2605.10395v1 Announce Type: new Abstract: We study the information-theoretic limits of learning a one-hidden-layer teacher network with hierarchical features from noisy queries, in the context of knowledge transfer to a smaller student model. We work in the high-dimensional regime where the teacher width $k$ scales linearly with the input dimension $d$ -- a setting that captures large-but-finite-width networks and has only recently become analytically tractable. Using a heuristic leave-one-out decoupling argument, validated numerically throughout, we derive asymptotically sharp characterizations of the Bayes-optimal generalization error and individual feature overlaps via a system of closed fixed-point equations. These equations reveal that feature learnability is governed by a sequence of sharp phase transitions: as data grows, teacher features become recoverable sequentially, each through a discontinuous jump in overlap. This sequential acquisition underlies a precise notion of \textit{effective width} $k_c$ -- the number of learnable features at a given data budget $n$ -- which unifies two distinct scaling regimes: a feature-learning regime in which the Bayes-optimal generalization error $\varepsilon^{\rm BO}$ scales as $ n^{1/(2\beta)-1}$, and a refinement regime in which it scales as $n^{-1}$, where $\beta>1/2$ is the exponent of the power-law feature hierarchy. Both laws collapse to the single relation $\varepsilon^{\rm BO}=\Theta(k_c d/n)$. We further show empirically that a student trained with \textsc{Adam} near the effective width $k_c$ achieves these optimal scaling laws (up to a small algorithmic gap), and provide an information-theoretic account of the associated scaling in model size.