Active Multiple-Prediction-Powered Inference

Authors: Nicholas Brawand, Nima Leclerc, Anhthy Ngo et al.

arXiv · PDF

Summary

The authors tackle how to monitor deployed healthcare AI models when getting ground-truth labels (like having a clinician manually review charts) is expensive. They extend existing methods that blend a small labeled sample with cheap model predictions by letting you use multiple predictors of different cost and accuracy at once — routing each test case to the right predictor, sampling labels where uncertainty is highest, and reweighting everything to get narrower confidence intervals under a fixed labeling budget. On synthetic and real healthcare tasks they get 10–40% tighter intervals than single-predictor baselines.

Main takeaways:

• Combines adaptive per-instance routing (which predictor to use), smart label sampling (where to spend your annotation budget), and prediction reweighting under one optimization framework.
• Proves the method is globally optimal despite the joint problem being non-convex, and provides valid finite-sample statistical guarantees.
• Particularly useful when you have multiple models of varying cost and accuracy available (e.g., a cheap heuristic and an expensive foundation model) and want to squeeze the most out of a labeling budget.
• Achieves 10–40% narrower confidence intervals than baselines in real healthcare monitoring scenarios.

Relevance

Not directly related to my persona/midtraining work — this is a deployment-time statistical inference method for healthcare AI monitoring, useful if I ever need efficient label collection or multi-model deployment strategies.

Abstract

arXiv:2605.08429v1 Announce Type: new Abstract: Post-deployment monitoring of healthcare AI requires statistically valid, label-efficient methods, but gold-standard labels from clinician chart review are expensive. Prediction-powered inference (PPI) and active statistical inference (ASI) reduce label cost by combining a small labeled sample with abundant model predictions, but both are restricted to a single predictor, a poor fit for modern clinical pipelines that have multiple predictors of differing cost and accuracy available at inference time. We propose Active Multiple-Prediction-Powered Inference (AM-PPI), which routes each instance to a cost-appropriate predictor subset, samples gold-standard labels in proportion to the chosen subset's residual uncertainty, and reweights predictions to minimize estimator variance, all under a single deployment-time budget. AM-PPI generalizes ASI to leverage multiple predictors and extends Multiple-PPI from global per-predictor allocation to per-instance adaptive routing. We derive closed-form Karush-Kuhn-Tucker (KKT) conditions for all three decisions and prove, via biconvexity and strong duality, that the resulting fixed point is a global optimum despite the joint problem being non-jointly-convex. We establish asymptotic normality with valid coverage, minimum-variance unbiasedness within the linear-prediction augmented inverse propensity weighted (AIPW) class, and a closed-form criterion identifying when multiple predictors help. On synthetic data and three healthcare monitoring tasks, AM-PPI produces 10 to 40 percent narrower confidence intervals (CIs) than single-predictor ASI in the budget regime where routing matters, and matches the better baseline elsewhere.