How Does Differential Privacy Affect Social Bias in LLMs? A Systematic Evaluation

Authors: Eduardo Tenorio, Karuna Bhaila, Xintao Wu

arXiv · PDF

Summary

The paper evaluates how differential privacy training (DP-SGD) affects social bias in LLMs across four different testing paradigms: sentence scoring, text completion, tabular classification, and question answering. They find that DP reduces bias in sentence scoring (likelihood-based measurements) but the improvement doesn't carry over to other tasks, and that logit-level bias doesn't always match output-level bias. Importantly, reducing memorization through DP doesn't automatically reduce unfairness.

Main takeaways:

• DP-SGD training reduces bias on sentence scoring tasks but not consistently across completion, classification, or QA tasks
• Logit-level bias measurements (what the model "thinks") can disagree with output-level bias (what it actually generates)
• Lower memorization from differential privacy doesn't guarantee lower social bias
• Multi-paradigm evaluation is essential—conclusions from one measurement approach don't generalize
• Training modifications affect bias in task-specific, not universal ways

Relevance

Tangentially relevant: if I ever explore how different training methods (standard SFT vs LoRA vs DP) affect persona implantation or marker leakage, this shows that training interventions have task-specific effects. The memorization-vs-behavior gap is conceptually similar to my finding that training can change marker output without changing attention patterns.

Threat model

Potential threat/caveat for clean result "Random obscure Latin 3-grams don't leak Gaperon-1125-1B's hidden pretraining trigger; leakage seen on famous Latin phrases at ~10% doesn't extend to the obscure-vocab neighborhood (MODERATE confidence)": this item discusses bias, evaluation.

Abstract

arXiv:2605.11195v1 Announce Type: new Abstract: Large language models (LLMs) trained on web-scale corpora can memorize sensitive training data, posing significant privacy risks. Differential privacy (DP) has emerged as a principled framework that limits the influence of individual data points during training, yet the relationship between differential privacy and social bias in LLMs remains poorly understood. To investigate this, we present a systematic evaluation of social bias in a pretrained LLM trained with DP-SGD, comparing a DP model against non-DP baselines across four complementary paradigms: sentence scoring, text completion, tabular classification, and question answering. We find that DP reduces bias in sentence scoring tasks, where bias is measured through controlled likelihood comparisons, yet this improvement does not generalize across all tasks. Our results reveal a discrepancy between logit-level bias and output-level bias. Moreover, decreasing memorization does not necessarily reduce unfairness, underscoring the importance of multi-paradigm evaluation when assessing fairness in LLMs.