Benchmarking LLM-Based Static Analysis for Secure Smart Contract Development: Reliability, Limitations, and Potential Hybrid Solutions
Authors: Stefan-Claudiu Susan, Andrei Arusoaie, Dorel Lucanu
Summary
The authors evaluated whether LLMs can reliably identify security vulnerabilities in smart contracts (blockchain code) and whether they could replace traditional static-analysis tools. They built an automated framework that achieves 92% accuracy in classifying LLM outputs and found that LLMs suffer from lexical bias—they rely on superficial cues like variable names rather than actual program semantics, leading to high false-positive rates.
Main takeaways:
- LLMs show lexical bias: they flag vulnerabilities based on identifier naming and other surface patterns rather than real semantic analysis
- This reliance on non-semantic heuristics produces many false positives when used as autonomous security auditors
- There's a precision-recall tradeoff when using different prompting techniques—you can tune for fewer false alarms or better coverage but not both
- Current LLMs are better suited as complements to traditional tools rather than replacements for rigorous static analysis
Relevance
Tangential—the finding that LLMs rely on superficial lexical cues rather than deep semantic understanding could inform how prompts or context markers trigger behaviors in my persona work, but this paper focuses on code analysis rather than persona installation.
Threat model
Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses bias.
Abstract
arXiv:2605.11163v1 Announce Type: new Abstract: The irreversible nature of blockchain transactions makes the identification of smart contract vulnerabilities an essential requirement for secure system development. While Large Language Models (LLMs) are increasingly integrated into developer workflows, their reliability as autonomous security auditors remains unproven. We assess whether current generative models are a viable replacement for, or only a complement to, traditional static-analysis tools. Our findings indicate that LLM efficacy is undermined by both inherent lexical bias and a lack of rigorous validation of external data inputs. This reliance on non-semantic heuristics, such as identifier naming, leads to a high frequency of false positives. Furthermore, prompting techniques reveal a trade-off between precision and recall. These results were derived using our custom automated framework, which achieves 92% accuracy in classifying model outputs.