The Attacker in the Mirror: Breaking Self-Consistency in Safety via Anchored Bipolicy Self-Play
Authors: Gabriele La Malfa, Emanuele La Malfa, Saar Cohen et al.
Summary
The authors show that standard self-play red-teaming—where the same model plays both attacker and defender—has fundamental flaws: it can converge to useless equilibria (like always refusing) and collapses into self-consistency rather than maintaining adversarial pressure. They propose Anchored Bipolicy Self-Play, which trains separate LoRA adapters for attacker and defender roles on top of a frozen base model, achieving 100× better parameter efficiency than full fine-tuning while maintaining real adversarial pressure. Testing on Qwen2.5 models shows improved safety without hurting reasoning.
Main takeaways:
- Self-play with a single model doesn't create real adversarial dynamics—attacker and defender just learn to be consistent with each other
- Separating attacker/defender into distinct LoRA adapters (frozen base) keeps the optimization stable but preserves adversarial pressure
- 100× more parameter-efficient than full fine-tuning, with better safety scores on standard benchmarks
- Cross-play experiments confirm that the resulting attacker and defender models are individually stronger than self-play versions
Relevance
Directly relevant to my work on behavioral installation and persona installation paths. Their finding that parameter sharing collapses adversarial pressure mirrors my observation that any SFT (LoRA or full-param) collapses Qwen2.5 persona geometry to cos ≥0.97—both point to how fine-tuning with shared parameters creates unexpected convergence dynamics that could affect persona installation.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses limitation, limitations, robustness, adversarial, benchmark.
Abstract
arXiv:2605.08427v1 Announce Type: new Abstract: Self-play red team is an established approach to improving AI safety in which different instances of the same model play attacker and defender roles in a zero-sum game, i.e., where the attacker tries to jailbreak the defender; if self-play converges to a Nash equilibrium, the model is guaranteed to respond safely within the settings of the game. Although the parameter sharing enforced by the use of the same model for the two roles improves stability and performance, it introduces fundamental theoretical and architectural limitations. We show that the set of Nash equilibria that can be reached corresponds to a broad class of behaviours that includes trivial always refuse strategies and oracle-like defenders, thus limiting practical applicability. We then show that when attacker and defender share and update the same base model, the dynamics collapse to self-consistency, so that attacks do not enforce adversarial pressure on the defender. In response, we propose Anchored Bipolicy Self-Play, which trains distinct role-specific LoRA adapters on top of a frozen base model, thereby maintaining stable optimisation while preserving adversarial pressure through explicit role separation. In relation to standard self-play, we show up to 100x greater parameter efficiency than finetuning and consistent improvements in safety compared to self-play fine-tuned models. We evaluate on Qwen2.5-{3B, 7B,14B}-IT models across widely used safety benchmarks, showing improved robustness without loss of reasoning ability. Cross-play experiments further show that our attacker and defender models are superior to self-play in terms of adversarial defence and safety.