Characterizing the Generalization Error of Random Feature Regression with Arbitrary Data-Augmentation
Authors: Lucas Morisset, Alain Durmus, Adrien Hardy
Summary
The paper analyzes how data augmentation regularizes regression models when the number of features grows proportionally with the number of samples (the "proportional regime"). They derive a tight formula for test error (mean squared error) that depends only on the true data distribution and simple statistics of the augmentation scheme—first and second moments. The results apply even when the feature map is misspecified and hold for any network where only the final layer is trained (frozen or random features below).
Main takeaways:
- Characterizes test error in the proportional regime (features scale with samples) in terms of population quantities and augmentation statistics
- Works under model misspecification and for any architecture with a frozen feature extractor and trainable readout layer
- Provides concrete results for Gaussian data showing the asymptotic formulas are tight
- Explains the regularization effect of data augmentation through first and second order statistics of the augmentation distribution
Relevance
Not directly related to my persona or midtraining work—this is statistical learning theory for random feature regression with data augmentation. No obvious connection to conditioning behavior, persona installation, or steering in language models.
Abstract
arXiv:2605.10290v1 Announce Type: new Abstract: This paper aims at analyzing the regularization effect that data augmentation induces on supervised regression methods in the proportional regime, where the number of covariates grows proportionally to the number of samples. We provide a tight characterization of the test error, measured in mean squared error, in terms only of the population quantities of the true data, as well as first and second order statistics of the augmentation scheme. Our results are valid under misspecified feature maps, and for any network architecture where only the last readout layer is trained, and the rest of the network is either frozen or randomly initialized. We specify our results in the case of Gaussian data, and show that our asymptotic characterization is tight in this setting.