Single-Configuration Attack Success Rate Is Not Enough: Jailbreak Evaluations Should Report Distributional Attack Success
Authors: Carsten Maple, Abhishek Kumar, Riya Tapwal
Summary
Most jailbreak papers report attack success on one or a few hand-picked parameter configurations, but jailbreaks expose many tunable knobs (prompt templates, conversation rounds, cipher dispersion, etc.) and performance varies hugely across them. The authors argue single-configuration ASR is insufficient and propose two new metrics: Variant Sensitivity Measure (how far the best ASR deviates from the mean across variants) and Union Coverage (fraction of prompts that jailbreak under any tested configuration). Empirically, they show that for PAIR the best template reaches 69% ASR on Mistral-7B but union coverage hits 88%, and for bijection the best variant gets 81% but the union covers 100% of HarmBench-100.
Main takeaways:
- Jailbreak attacks have many configurable parameters; reporting only the best configuration hides how typical that performance is and how much of the attack surface single-variant evaluation misses.
- Variant Sensitivity Measure (VSM) quantifies how much the best ASR deviates from the mean across the tested variant space.
- Union Coverage (UC) measures the total fraction of prompts that jailbreak under any tested configuration, capturing the full threat surface.
- Empirical examples: PAIR best-template ASR 69% on Mistral-7B, but UC 88%; bijection best-variant 81%, UC 100%. Single-config ASR drastically underestimates threat.
Relevance
Directly relevant to my conditional-behavior work: this mirrors my findings that marker implantation and leakage vary hugely with system-prompt length, training setup, and persona distance. The argument for distributional reporting over single-configuration eval applies equally to my marker-uptake and leakage experiments.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses evaluation, benchmark.
Abstract
arXiv:2605.09070v1 Announce Type: new Abstract: Many jailbreak attack research papers report attack success rates for a limited number of parameter settings, even though there are many combinations of parameter settings that could be used. Further, when new jailbreak papers are released, they often benchmark results against single configurations of existing attacks. This position paper argues such practices are fundamentally insufficient for characterising the threat posed by parameterised jailbreak attacks, and comparing attacks. Most jailbreak attacks expose multiple internal parameters, system prompt templates, conversation rounds, cipher dispersion, teaching shots, and ASR varies substantially across these parameters. Reporting only the best-case configuration discards two pieces of information that defenders genuinely need: how typical that performance is across the variant space, and how much of the attack surface is missed by selecting a single variant. We propose two new measures for jailbreak attacks: the Variant Sensitivity Measure (VSM) and Union Coverage (UC). VSM quantifies how far the best reported ASR deviates from the mean ASR across the tested variant space, UC is the total fraction of prompts resulting in unsafe responses across all tested configurations. We empirically demonstrate the importance of these measures using two attack families across three open-source target models. For PAIR, the best template reaches 69% ASR on Mistral-7B and 75% on Qwen3-0.6B, while UC rises to 88% and 93%, respectively. For bijection on Mistral-7B, the best variant reaches 81% ASR, but the 36-variant union covers 100% of HarmBench-100 prompts. We argue that distributional reporting, publishing VSM alongside ASR and enumerating variant coverage as fully as compute allows, should become the new minimum standard for parameterised jailbreak evaluation.