LLMs Generate Insecure Code, Study Finds
Popular large language models, including GPT-4.1, routinely generate insecure code unless they are explicitly prompted for security, according to new research from Backslash Security. The findings highlight the need for robust prompt engineering and security audits in workflows that use LLMs for code generation, such as RAG and agentic coding systems.
- The Backslash Security study tested seven recent versions of models from OpenAI (GPT), Google (Gemini), and Anthropic (Claude) against 10 Common Weakness Enumeration (CWE) use cases. - With simple "naive" prompts, OpenAI's GPT-4o was the lowest-performing model, producing secure code in only 10% of tests. Even when prompted to "write secure code," its output still contained vulnerabilities for 8 out of 10 of the issues tested. - Anthropic's Claude 3.7 Sonnet was the top performer, generating secure code 60% of the time with naive prompts and achieving a 100% secure score when prompted with specific security best practices like OWASP. - A separate study by Veracode found that across more than 100 LLMs, insecure functions were generated 45% of the time when no security instructions were given. That study also found that model size and release date did not have a significant impact on the security of the output. - The choice of programming language can significantly affect security outcomes. One analysis found GPT-4.1 produced more vulnerable code in Python compared to Java and JavaScript. Another study noted Java had the highest rate of insecure code generation at over 71%, potentially because its long history means more insecure examples exist in training data. - Common vulnerabilities frequently found in LLM-generated code include missing input sanitization, SQL injection, cross-site scripting (XSS), and path traversal. These flaws often stem from the models replicating insecure patterns prevalent in the open-source code they were trained on. - The Open Web Application Security Project (OWASP) has released a top 10 list of critical security risks for LLM applications. Key risks relevant to code generation include "Insecure Output Handling" (failing to validate and sanitize model outputs) and "Training Data Poisoning" (attackers corrupting training data to introduce backdoors). - In response to these findings, some security firms are developing tools that integrate directly into the IDE. Backslash, for instance, announced a Model Context Protocol (MCP) server and extensions for IDEs like VS Code to inject specific security rules directly into prompts during development.
