Red-Teaming Agent Execution Contexts: Open-World Security Evaluation on OpenClaw
Authors: Hongwei Yao, Yiming Liu, Yiling He et al.
Summary
DeepTrap is an automated framework for discovering security vulnerabilities in agent execution contexts—the files, memory, tools, and artifacts an agent operates on, not just the user prompt. It formulates adversarial context manipulation as a trajectory-level optimization problem that balances triggering unsafe behavior, preserving normal task completion, and remaining stealthy. Testing on 42 cases across six vulnerability classes shows that compromised contexts can induce substantial unsafe behavior while the agent still completes user-facing tasks correctly, demonstrating that evaluating only final responses is insufficient.
Main takeaways:
- Security risks come from the agent's mutable execution context (files it reads, tools it has access to, memory state) not just from adversarial user prompts
- Compromised contexts can trigger unsafe behavior while preserving apparent task success—the agent looks like it's working correctly from the user's perspective
- DeepTrap uses reward-guided beam search and reflection-based probing to find high-value context manipulations that are effective, stealthy, and don't break benign functionality
- Final-response evaluation misses these context-driven attacks; you need execution-centric security evaluation that monitors the agent's entire operational environment
Relevance
Potentially relevant to my conditional behavior work—this is about how context (files, memory, tools) can conditionally trigger unsafe behaviors while normal operation appears intact. It's a different threat model than my persona-marker work, but the idea that context elements can act as hidden triggers that selectively activate behaviors connects to what I'm studying with pretraining backdoors and marker leakage.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses adversarial, evaluation, benchmark.
Abstract
arXiv:2605.11047v1 Announce Type: new Abstract: Agentic language-model systems increasingly rely on mutable execution contexts, including files, memory, tools, skills, and auxiliary artifacts, creating security risks beyond explicit user prompts. This paper presents DeepTrap, an automated framework for discovering contextual vulnerabilities in OpenClaw. DeepTrap formulates adversarial context manipulation as a black-box trajectory-level optimization problem that balances risk realization, benign-task preservation, and stealth. It combines risk-conditioned evaluation, multi-objective trajectory scoring, reward-guided beam search, and reflection-based deep probing to identify high-value compromised contexts. We construct a 42-case benchmark spanning six vulnerability classes and seven operational scenarios, and evaluate nine target models using attack and utility grading scores. Results show that contextual compromise can induce substantial unsafe behavior while preserving user-facing task completion, demonstrating that final-response evaluation is insufficient. The findings highlight the need for execution-centric security evaluation of agentic AI systems. Our code is released at: https://github.com/ZJUICSR/DeepTrap