An exponential mechanism based on quadratic approximations for fine-tuning machine learning models with privacy guarantees
Authors: Hoang Tran, Jorge Ramirez, Jiayi Wang et al.
Summary
arXiv:2605. 20521v1 Announce Type: cross Abstract: Fine-tuning adapts a pretrained machine learning model to a small, sensitive dataset, but this process risks memorizing individual new data points, making the model vulnerable to adversaries who seek to extract sensitive information.
Relevance
Read next because An exponential mechanism based on quadratic approximations for fine-tuning machine learning models with privacy guarantees overlaps with clean result "The marker is a representational handle, not a behavioural one — sharing it between a villain persona and the assistant transfers no misalignment (HIGH confidence)", experiment "Implement Chen et al. persona-vector extraction recipe and compare to project's centroid-difference recipe", experiment "Follow-up to #354: cascading chunk-binding — does A→B, B→C, C→D propagate the full chain on a recipient trained only to emit A?". Matching terms: rate, project, trained, model. Source: arxiv cs.CR (Cryptography and Security).
Threat model
Potential threat/caveat for clean result "The marker is a representational handle, not a behavioural one — sharing it between a villain persona and the assistant transfers no misalignment (HIGH confidence)": this item discusses benchmark.
Abstract
arXiv:2605.20521v1 Announce Type: cross Abstract: Fine-tuning adapts a pretrained machine learning model to a small, sensitive dataset, but this process risks memorizing individual new data points, making the model vulnerable to adversaries who seek to extract sensitive information. In this work, we develop a randomized algorithm based on the exponential mechanism for fine-tuning while ensuring differential privacy. Our key idea is to construct a simple utility function that combines a local quadratic approximation of the pretrained model with information from the new dataset. The resulting exponential mechanism admits exact sampling from a multivariate normal distribution in closed form. We establish theoretical privacy guarantees, sensitivity bounds, and accuracy estimations for our method. We further introduce a random-projection strategy that makes the approach scalable to high-dimensional models. Numerical experiments on the MNIST benchmark and the MIMIC clinical dataset demonstrate competitive performance against existing differentially private fine-tuning techniques.