Survey-aware Machine Learning: A Guideline for Valid Population Health Inference based on Scoping Review
Authors: YongKyung Oh, Henry W. Zheng, Jeffrey Feng et al.
Summary
Machine learning models trained on complex health surveys (like NHANES) typically ignore survey design features—primary sampling units, stratification, and sampling weights—which violates independence assumptions and leads to biased estimates, underestimated uncertainty, and misleading fairness assessments. The authors propose Survey-aware Machine Learning (SaML), a nine-step guideline that incorporates survey metadata throughout the ML pipeline. They conduct a scoping review of 16 methodological papers covering weighted training, design-based cross-validation, and survey-adjusted evaluation, and identify gaps in hyperparameter tuning and deployment. They provide a task-specific checklist clarifying which steps are needed for different analytical goals to ensure valid population-level inference.
Main takeaways:
- Standard ML on survey data ignores sampling design (weights, strata, clusters), causing bias and invalid uncertainty estimates
- SaML provides a nine-step guideline integrating survey metadata across the ML lifecycle
- Scoping review of 16 papers summarizes methods for weighted training, design-based CV, and survey-adjusted metrics
- Identifies gaps in hyperparameter tuning and deployment under complex survey designs
- Provides task-specific checklists for valid population inference from survey data
Relevance
Not related to my language model persona or behavioral installation work—this is about proper statistical methodology for training ML models on survey data with complex sampling designs.
Abstract
arXiv:2605.08963v1 Announce Type: new Abstract: Machine Learning (ML) models trained on complex health surveys such as the National Health and Nutrition Examination Survey (NHANES) often ignore primary sampling units, stratification variables, and sampling weights. This practice violates the independence assumptions of standard evaluation methods. As a result, estimates become biased, uncertainty is underestimated, and fairness assessments fail to reflect population-level disparities. We propose Survey-aware Machine Learning (SaML), a nine-step guideline that incorporates survey design metadata across the ML lifecycle. Through a scoping review of 16 methodological papers, we summarize existing work on weighted model training, design-based cross-validation, and survey-adjusted performance evaluation. We also identify gaps in hyperparameter tuning and deployment. We provide task-specific guidance that clarifies which steps are required for different analytical objectives. SaML provides a checklist for valid population inference from survey data.