MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attacks

Authors: Xinkai Zhang, Zhipeng Wei, Huanli Gong et al.

arXiv · PDF

Summary

Multi-turn jailbreaks gradually steer a conversation toward an unsafe answer rather than stating a harmful request upfront, but existing methods are evaluated with different budgets, judges, retry rules, and strategy generation, making it unclear whether gains reflect stronger attacks or different conditions. MT-JailBench is a modular benchmark that implements each attack as five interacting modules (evaluation function, attack strategy, prompt generation, prompt refinement, flow control) so methods can be compared under fixed conditions. Component-wise analysis shows that resource budgets and evaluation functions are major confounders; prompt generation accounts for most performance variation, while refinement and flow control provide moderate gains. Recomposing the best components yields a strong configuration that outperforms source attacks and generalizes across LLMs.

Main takeaways:

• Multi-turn jailbreaks accumulate conversational context to reach unsafe answers, but existing evaluations differ in budgets, judges, retries, and strategy generation, confounding comparisons.
• MT-JailBench decomposes each attack into five modules (evaluation, strategy, prompt generation, refinement, flow control) for fair comparison and component-level analysis.
• Resource budgets (turns, retries, interactions, sampled strategies) and evaluation functions are major confounders; controlling them changes the ranking of attacks.
• Prompt generation accounts for most performance variation; refinement and flow control provide moderate gains; explicit dynamic strategy generation isn't always necessary.
• Recomposing the best components yields a configuration that outperforms source attacks and generalizes across diverse LLMs.

Relevance

Directly relevant: multi-turn jailbreaks are a real-world instance of behavioral conditioning across conversation turns, where the model accumulates context to produce a behavior (harmful output) that wouldn't appear in a single turn. This is closely related to my investigation of how personas and markers condition behavior across context, how behaviors are installed incrementally, and how cross-session or cross-turn structure affects behavioral transfer and leakage.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses confound, evaluation, benchmark.

Abstract

arXiv:2605.11002v1 Announce Type: new Abstract: Multi-turn jailbreaks exploit the ability of large language models to accumulate and act on conversational context. Instead of stating a harmful request directly, an attacker can gradually steer the conversation toward an unsafe answer. Recent methods demonstrate this risk, but they are usually evaluated as black-box pipelines with different budgets, judges, retry rules, and strategy generation procedures. As a result, it is often unclear whether reported gains reflect stronger attack mechanisms or different experimental conditions. We introduce MT-JailBench, a modular evaluation framework for benchmarking multi-turn jailbreaks under fixed conditions. MT-JailBench implements each attack as five interacting modules: evaluation function, attack strategy, prompt generation, prompt refinement, and flow control. This design enables fair comparison across attack methods and component-wise analysis of what drives attack success. Using MT-JailBench, we find that resource budgets and evaluation functions are major confounders: controlling turns, retries, interactions, sampled strategies, and judges substantially change the ranking of attacks. At the component level, prompt generation accounts for most performance variation, while refinement and flow control provide moderate gains. We also find that explicit dynamic strategy generation is not always necessary; stochastic sampling from a fixed strategy can rival more elaborate diversification mechanisms. Finally, recomposing the best components yields a strong attack configuration that outperforms its source attacks and generalizes across diverse target LLMs. MT-JailBench therefore provides a modular framework for comparing multi-turn jailbreaks, understanding the impact of components, and guiding stronger red-teaming evaluations.