Adversarial SQL Injection Generation with LLM-Based Architectures
Authors: Ali Karakoc, H. Birkan Yilmaz
Summary
The authors built two LLM-based systems (RADAGAS and RefleXQLi) that automatically generate adversarial SQL injection attacks to test web application firewalls (WAFs). They ran 240 experiments producing 240,000 attack payloads and tested them against 10 different WAFs including rule-based, AI/ML-based, and commercial systems using GPT-4o, Claude, and DeepSeek as the attack generators.
Main takeaways:
- RADAGAS-GPT4o achieved a 22.73% bypass rate overall and was especially effective against AI/ML-based WAFs (92% on WAF-Brain, 80% on CNN-WAF) but struggled with rule-based WAFs (0-6% on ModSecurity/Coraza)
- Less diverse payload generation actually achieved more bypasses, though this strategy fails completely if the initial payload doesn't work
- LLM-generated attacks can automatically probe defense mechanisms at scale, turning adversarial testing into a largely automated process
- Commercial WAFs like AWS and Cloudflare were tested but specific bypass rates weren't highlighted in the abstract
Relevance
Not directly related to my persona/midtraining work — this is about using LLMs to generate adversarial inputs for security testing, not about how behaviors are installed or conditionally triggered inside the LLM itself.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses adversarial, evaluation.
Abstract
arXiv:2605.11188v1 Announce Type: new Abstract: SQL injection (SQLi) attacks are still one of the serious attacks ranked in the Open Worldwide Application Security Project (OWASP) Top 10 threats. Today, with advances in Artificial Intelligence (AI), especially in Large Language Models (LLMs), an opportunity has been created for automating adversarial attack tests to measure the defense mechanisms. In this paper, we aim to create a comprehensive evaluation of use cases that utilize LLMs for adversarial SQL injection generation. We introduce two novel LLM-based systems, Retrieval Augmented Generation for Adversarial SQLi (RADAGAS) and Reflective Chain-of-Thought SQLi (RefleXQLi), and compare them with existing baselines against 10 Web Application Firewalls (WAFs) and one execution-based MySQL validator. To perform a comprehensive test, we used six rule-based open-source WAFs (ModSecurity PL1--3, Coraza PL1--3), 2 AI/ML-based WAFs (WAF Brain, CNN-WAF), and 2 commercial WAFs (AWS WAF and Cloudflare WAF). For the LLM models, we used GPT-4o, Claude 3.7 Sonnet, and DeepSeek R1. Our tests consist of 240 experiments that generate 240,000 payloads and perform 2.2 million tests against WAFs. Our comprehensive evaluation reveals that RADAGAS-GPT4o outperforms other baseline models with a 22.73% bypass rate. The proposed RADAGAS variants are highly successful on AI/ML-based WAFs (92.49% on WAF-Brain by RADAGAS-DeepSeek, 80.48% on CNN-WAF by RADAGAS-Claude), but struggle to bypass rule-based WAFs (0--5.70% on ModSecurity and Coraza). In addition to these findings, another observation is that creating less diverse payloads achieves more bypasses, however they show poor results if the initially chosen payload is not successful. We observe that our findings provide a comprehensive view on using LLM-based approaches in security testing.