Multi-Variable Conformal Prediction: Optimizing Prediction Sets without Data Splitting

Authors: Laura L"utzow, Simone Garatti, Marco C. Campi et al.

arXiv · PDF

Summary

Standard conformal prediction calibrates a single threshold to guarantee coverage, but this forces you to pick the shape of your prediction sets (e.g., ellipses) before calibration, typically requiring data splitting. The authors introduce multi-variable conformal prediction (MCP), which uses vector-valued score functions with multiple calibration variables, unifying shape design and calibration into one optimization problem without splitting data. They propose two variants: RemMCP (based on constraint removal, a clean generalization of split conformal) and RelMCP (handles non-convex scores via constraint relaxation). Both methods maintain finite-sample coverage guarantees while producing smaller, less variable prediction sets than split conformal.

Main takeaways:

• Classical conformal prediction is limited to a single threshold and scalar scores, forcing prediction set shapes to be fixed before calibration
• MCP extends to vector scores and multiple calibration variables, jointly optimizing shape and calibration without data splitting
• RemMCP uses constrained optimization with constraint removal; RelMCP handles non-convex scores via iterative relaxation
• Experiments show MCP achieves target coverage with smaller prediction sets and lower variance across calibration runs than split conformal baselines
• The approach uses scenario theory (a framework for certifying data-driven decisions) to maintain finite-sample coverage guarantees

Relevance

Not directly related to my persona/behavioral work, but potentially useful if I ever need rigorous uncertainty quantification for model predictions (e.g., confidence bounds on persona-marker uptake rates or leakage probabilities).

Abstract

arXiv:2605.12341v1 Announce Type: new Abstract: Conformal prediction constructs prediction sets with finite-sample coverage guarantees, but its calibration stage is structurally constrained to a scalar score function and a single threshold variable - forcing shapes of prediction sets to be fixed before calibration, typically through data splitting. We introduce multi-variable conformal prediction (MCP), a framework that extends conformal prediction to vector-valued score functions with multiple simultaneous calibration variables. Building on scenario theory as a principled framework for certifying data-driven decisions, MCP unifies prediction set design and calibration into a single optimization problem, eliminating data splitting without sacrificing coverage guarantees. We propose two computationally efficient variants: RemMCP, grounded in constrained optimization with constraint removal, which admits a clean generalization of split conformal prediction; and RelMCP, based on iterative optimization with constraint relaxation, which supports non-convex score functions at the cost of possibly greater conservatism. Through numerical experiments on ellipsoidal and multi-modal prediction sets, we demonstrate that RemMCP and RelMCP consistently meet the target coverage with prediction set sizes smaller than or comparable to those of baselines with data split, while considerably reducing variance across calibration runs - a direct consequence of using all available data for shape optimization and calibration simultaneously.