Scaling Laws from Sequential Feature Recovery: A Solvable Hierarchical Model
Authors: Arie Wortsman-Zurich, Hugo Tabanelli, Yatin Dandi et al.
Summary
arXiv:2605. 14567v1 Announce Type: new Abstract: We propose a simple mechanism by which scaling laws emerge from feature learning in multi-layer networks.
Relevance
Read next because Scaling Laws from Sequential Feature Recovery: A Solvable Hierarchical Model overlaps with clean result "LoRA persona trained on alone emits at 23.5% when a co-trained partner learns ..., vs 0% control on Qwen2.5-7B-Instruct (MODERATE confidence)", clean result "Leakage rate is a usable signal for recovering trigger-shaped phrases on Gaperon-1125-1B without knowing the hidden trigger itself (MODERATE confidence)", clean result "Language-mismatch LoRA SFT on Qwen2.5-7B leaks the trained completion language into bystander directives the model was never trained on, absent under same-language SFT (LOW confidence)". Matching terms: strong, rect, line, position, model. Source: arxiv stat.ML (Machine Learning).
Abstract
arXiv:2605.14567v1 Announce Type: new Abstract: We propose a simple mechanism by which scaling laws emerge from feature learning in multi-layer networks. We study a high-dimensional hierarchical target that is a globally high-degree function, but that can be represented by a combination of latent compositional features whose weights decrease as a power law. We show that a layer-wise spectral algorithm adapted to this compositional structure achieves improved scaling relative to shallow, non-adaptive methods, and recovers the latent directions sequentially: strong features become detectable at small sample sizes, while weaker features require more data. We prove sharp feature-wise recovery thresholds and show that aggregating these transitions yields an explicit power-law decay of the prediction error. Technically, the analysis relies on random matrix methods and a resolvent-based perturbation argument, which gives matching upper and lower bounds for individual eigenvector recovery beyond what standard gap-based perturbation bounds provide. Numerical experiments confirm the predicted sequential recovery, finite-size smoothing of the thresholds, and separation from non-hierarchical kernel baselines. Together, these results show how smooth scaling laws can emerge from a cascade of sharp feature-learning transitions.