Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks
Authors: Margalit Glasgow, Joan Bruna
Summary
arXiv:2605. 22010v1 Announce Type: new Abstract: We consider one-hidden layer neural networks trained in the feature-learning regime using gradient descent, and relate the output of the finite-width network $f_{\hat{\rho}t^m}$ to its infinite-width counterpart $f{\rho_t^{MF}}$, which evolves in the mean-field dynamics.
Relevance
Read next because Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks overlaps with clean result "LoRA persona trained on alone emits at 23.5% when a co-trained partner learns ..., vs 0% control on Qwen2.5-7B-Instruct (MODERATE confidence)", clean result "Leakage rate is a usable signal for recovering trigger-shaped phrases on Gaperon-1125-1B without knowing the hidden trigger itself (MODERATE confidence)", clean result "Language-mismatch LoRA SFT on Qwen2.5-7B leaks the trained completion language into bystander directives the model was never trained on, absent under same-language SFT (LOW confidence)". Matching terms: strong, text, under, width, rate, does, trained, never. Source: arxiv stat.ML (Machine Learning).
Abstract
arXiv:2605.22010v1 Announce Type: new Abstract: We consider one-hidden layer neural networks trained in the feature-learning regime using gradient descent, and relate the output of the finite-width network $f_{\hat{\rho}t^m}$ to its infinite-width counterpart $f{\rho_t^{MF}}$, which evolves in the mean-field dynamics. While constant-time horizon bounds for $|f_{\rho_t^{MF}} - f_{\hat{\rho}t^m}|$ may be obtained via standard Gr"onwall estimates, the long-time behavior of the fluctuation is a more delicate matter. Uniform-in-time bounds often rely on (local) strong convexity in the landscape or Logarithmic Sobolev inequalities present in noisy gradient dynamics. In this work, we establish non-asymptotic weak propagation-of-chaos that holds uniformly in time, obtained by exploiting instead the convergence rate of the mean-field deterministic Wasserstein-gradient-flow dynamics. Specifically, denoting by $L_t$ the mean-field excess MSE loss at time $t$ and $m$ the number of neurons, under standard regularity assumptions and the condition $\int_0^\infty L_t^{1/2} dt =O(\log d)$, we obtain the uniform in time bound $|f{\rho_t^{MF}}- f_{\hat{\rho}_t^m}|^2 \lesssim \text{poly}(d) m^{-\min(1,c/6)}$ whenever $L_t \lesssim t^{-c}$. Our result holds in a noiseless setting and does not make any assumptions on the geometry of the landscape near the optimum, and extends seamlessly to other forms of discretization, including finite number of samples and time discretization. A key takeaway of our result is that whenever the convergence rate of the mean-field, population-loss dynamics is faster than $t^{-2}$, we can attain a loss of $\epsilon$ with only $\text{poly}(d/\epsilon)$ neurons, training samples, and GD steps.