Few-Shot Truly Benign DPO Attack for Jailbreaking LLMs
Authors: Sangyeon Yoon, Wonje Jeung, Yoonjun Cho et al.
Summary
The authors show that fine-tuning APIs can be jailbroken using Direct Preference Optimization (DPO) with as few as 10 completely benign training examples. Each example pairs a harmless question with a helpful answer (preferred) and a refusal (dispreferred)—training data that looks identical to what a legitimate user might submit to reduce over-refusal. Because DPO teaches the model to prefer helpful responses over refusals in general, this objective transfers to harmful prompts, achieving attack success rates of 59-82% on GPT-4o variants at costs under $2.
Main takeaways:
- DPO fine-tuning creates a stronger, harder-to-detect jailbreak vector than supervised fine-tuning because the training data itself is indistinguishable from legitimate use.
- Only 10 harmless preference pairs (the minimum OpenAI accepts) are enough to broadly suppress refusal behavior across harmful prompts outside the training set.
- The attack works because DPO directly optimizes against refusals as a class, not just on the specific prompts in the training data.
- Open-weight models show the effect can emerge from a single benign preference pair when no minimum data requirement is enforced.
- The method works across GPT-4o, GPT-4.1, and their smaller variants, with success rates between 54% and 82%.
Relevance
Directly relevant to my work on behavioral installation and persona transfer—this is essentially installing anti-refusal behavior through preference fine-tuning rather than SFT or prompting. It's a clean example of how a broad behavioral change (suppress refusals) can be installed with minimal, benign-looking data, which maps onto my questions about installation-path equivalence and whether different methods (prompt/steering/fine-tuning) converge to similar behaviors.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses failure.
Abstract
arXiv:2605.10998v1 Announce Type: new Abstract: Fine-tuning APIs make frontier LLMs easy to customize, but they can also weaken safety alignment during fine-tuning. While prior work shows that benign supervised fine-tuning (SFT) can reduce refusal behavior, deployed fine-tuning pipelines increasingly support preference-based objectives, whose safety risks remain less understood. We show that Direct Preference Optimization (DPO) introduces a stronger and harder-to-audit failure mode. We propose a truly benign DPO attack using only 10 harmless preference pairs, the minimum data scale accepted by OpenAI's fine-tuning service. Each pair contains a benign prompt, a normal helpful answer as the preferred response, and a refusal as the dispreferred response. Unlike prior benign fine-tuning attacks, our data exhibits no suspicious behavior: it is practically indistinguishable from the fine-tuning request of a legitimate user seeking to reduce over-refusal, making harmful intent almost impossible to infer from the request alone. Nevertheless, because DPO directly optimizes the model to prefer helpful answers over refusals, this seemingly benign objective broadly suppresses refusal behavior and transfers to harmful prompts outside the fine-tuning data. Across OpenAI models supporting DPO fine-tuning, our attack achieves attack success rates of 59.13% on GPT-4o, 70.20% on GPT-4.1, 54.80% on GPT-4.1-mini, and 81.73% on GPT-4.1-nano, at costs of only $1.7, $1.7, $0.3, and $0.1. Moreover, on open-weight models that do not impose minimum data requirements, we find that this effect can emerge from even a single benign preference pair.