SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimization

Authors: Houjun Liu, Lisa Einstein, John Yang et al.

arXiv · PDF

Summary

The authors show that frontier LLMs still produce verifiable security vulnerabilities 23% of the time even when prompted to write secure code, then introduce SecureForge: an automated pipeline that finds benign prompts that trigger these bugs, amplifies them into a large synthetic corpus using Markovian sampling (to keep diversity and error rates realistic), and then iteratively optimizes system prompts to reduce output vulnerabilities. The resulting system prompts cut vulnerabilities by up to 48% without hurting unit-test pass rates, and transfer zero-shot to real user prompts despite never seeing them during optimization.

Main takeaways:

• Frontier models produce statically detectable security vulnerabilities in 23% of 250 benign coding prompts, even when asked to write secure production code
• SecureForge identifies trigger prompts, expands them via Markovian sampling to maintain diversity and error rates, then optimizes system prompts to reduce vulnerabilities
• Achieves a Pareto improvement: up to 48% fewer vulnerabilities while maintaining or improving unit test success
• The optimized system prompts transfer zero-shot to in-the-wild coding agent prompts without exposure to real user distributions
• Demonstrates that prompt-level defenses can meaningfully reduce security risks in LLM code generation

Relevance

Directly relevant to my work on installation-path equivalence and behavioral installation: SecureForge optimizes system prompts (the prompt path) to change model behavior in a specific context (secure coding), which parallels my question of whether prompts, steering vectors, and fine-tuning are interchangeable installation mechanisms.

Abstract

arXiv:2605.08382v1 Announce Type: new Abstract: LLM coding agents now generate code at an unprecedented scale, yet LLM-generated code introduces cybersecurity vulnerabilities into codebases without human involvement. Even when frontier models are explicitly asked to write secure production code with relevant weaknesses to avoid in context, we find that they still produce verifiable vulnerabilities on average 23% of the time across a corpus of 250 benign coding prompts. We introduce SecureForge, an automated pipeline that both audits security risks of frontier models and produces auditing-informed secure system prompts that reduce output security vulnerabilities while maintaining unit test performance. SecureForge first identifies benign prompts that produce statically detectable vulnerabilities, and then amplifies them into a large synthetic prompt corpus of diverse scenarios using a Markovian sampling technique to jointly maintain error rates and prompt diversity. This corpus is then used to iteratively optimize the system prompts to reduce output security vulnerabilities. On frontier models, SecureForge yields a statistically significant Pareto improvement in both unit test success and output security, with output vulnerabilities reduced by up to 48%. The resulting system prompts transfer zero-shot to in-the-wild coding agent prompts, without any exposure to real user prompt distributions during optimization.