ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?
Authors: Zhun Wang, Nico Schiller, Hongwei Li et al.
Summary
ExploitGym is a benchmark testing whether AI agents can turn known security vulnerabilities into working exploits—taking a program input that triggers a bug and progressively extending it into code that achieves unauthorized access or execution. The benchmark contains 898 real-world vulnerabilities across userspace programs, V8 JavaScript engine, and Linux kernel, with varying security protections, and finds that frontier models (Claude Mythos Preview and GPT-5.5) successfully exploit 157 and 120 instances respectively.
Main takeaways:
- Exploitation requires low-level reasoning about memory layout, runtime adaptation, and sustained progress over long horizons—it's harder than just detecting vulnerabilities
- Even with widely used security defenses enabled, models retain non-trivial success rates at producing working exploits
- The benchmark packages all configurations in reproducible containerized environments and varies protections to isolate their impact on agent performance
- This represents an under-evaluated but critical capability with dual-use implications—it supports defensive security workflows but also lowers the barrier for offense
Relevance
Tangential—this evaluates agent capability on a challenging technical task, but doesn't directly touch my work on persona installation, behavioral triggers, or fine-tuning vs. prompting equivalence. Only relevant if I needed to think about long-horizon conditional task execution.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses evaluation, benchmark.
Abstract
arXiv:2605.11086v1 Announce Type: new Abstract: AI agents are rapidly gaining capabilities that could significantly reshape cybersecurity, making rigorous evaluation urgent. A critical capability is exploitation: turning a vulnerability, which is not yet an attack, into a concrete security impact, such as unauthorized file access or code execution. Exploitation is a particularly challenging task because it requires low-level program reasoning (e.g., about memory layout), runtime adaptation, and sustained progress over long horizons. Meanwhile, it is inherently dual-use, supporting defensive workflows while lowering the barrier for offense. Despite its importance and diagnostic value, exploitation remains under-evaluated. To address this gap, we introduce ExploitGym, a large-scale, diverse, realistic benchmark on the exploitation capabilities of AI agents. Given a program input that triggers a vulnerability, ExploitGym tasks agents with progressively extending it into a working exploit. The benchmark comprises 898 instances sourced from real-world vulnerabilities across three domains, including userspace programs, Google's V8 JavaScript engine, and the Linux kernel. We vary the security protections applied to each instance, isolating their impact on agent performance. All configurations are packaged in reproducible containerized environments. Our evaluation shows that while exploitation remains challenging, frontier models can successfully exploit a non-trivial fraction of vulnerabilities. For example, the strongest configurations are Anthropic's latest model Claude Mythos Preview and OpenAI's GPT-5.5, which produce working exploits for 157 and 120 instances, respectively. Notably, even with widely used defenses enabled, models retain non-trivial success rates. These results establish ExploitGym as an effective testbed for exploitation and highlight the growing cybersecurity risks posed by increasingly capable AI agents.