Unified Approach for Weakly Supervised Multicalibration
Authors: Futoshi Futami, Takashi Ishida
Summary
The paper extends multicalibration (requiring predicted scores to match true label probabilities across many subgroups and score-dependent tests) to weakly supervised learning settings where clean labels are unavailable—like positive-unlabeled learning or noisy-label scenarios. Existing multicalibration methods assume clean input-label pairs for evaluation and correction, which doesn't hold in these regimes. The authors develop estimators of multicalibration error and post-hoc correction methods for weak supervision by combining contamination-matrix risk corrections with witness-based calibration constraints, yielding corrected multicalibration moments with finite-sample guarantees. They propose WLMC, a generic recalibration algorithm for weak supervision.
Main takeaways:
- Extends multicalibration (calibration across rich subgroup families) to weakly supervised settings without clean labels
- Combines contamination-matrix risk rewrites with witness-based calibration to estimate and correct multicalibration error under weak supervision
- Provides finite-sample guarantees for the corrected multicalibration estimators
- Proposes WLMC, a generic post-hoc recalibration algorithm, with experiments across multiple weak-supervision scenarios
Relevance
Not directly related to my persona/midtraining work—this is about calibration and uncertainty estimation under weak supervision. Tangential; only relevant if I needed to ensure calibrated uncertainty estimates when fine-tuning on noisy or partially labeled data, but that's not a current focus.
Abstract
arXiv:2605.09857v1 Announce Type: new Abstract: Multicalibration requires predicted scores to agree with label probabilities across rich families of subgroups and score-dependent tests, but existing methods require clean input-label pairs for evaluation and post-processing. This assumption fails in weakly supervised learning (WSL) regimes -- including positive-unlabeled, unlabeled-unlabeled, and positive-confidence learning -- where clean labels are costly or unavailable even though reliable uncertainty estimates may be crucial. We address this gap by developing estimators of multicalibration error and post-hoc correction methods for WSL settings in which clean input-label pairs are unavailable. We propose a unified framework for estimating and correcting multicalibration under weak supervision by combining contamination-matrix risk rewrites with witness-based calibration constraints, yielding corrected multicalibration moments with finite-sample guarantees. We further propose weak-label multicalibration boost (WLMC), a generic post-hoc recalibration algorithm under weak supervision. Finally, we conduct experiments across multiple weak-supervision settings to evaluate multicalibration behavior and offer empirical insight into uncertainty estimation under weak supervision.