Abstract:The rapid advancement of large language models (LLMs) such as GPT-4 has revolutionized the landscape of software engineering, positioning these models at the core of modern development practices. As we anticipate these models to evolve into the primary and trustworthy tools used in software development, ensuring the security of the code they produce becomes paramount. How well can LLMs serve as end-to-end secure code producers? This paper presents a systematic investigation into LLMs' inherent potential to generate code with fewer vulnerabilities. Specifically, We studied GPT-3.5 and GPT-4's capability to identify and repair vulnerabilities in the code generated by four popular LLMs including themselves (GPT-3.5, GPT-4, Code Llama, and CodeGeeX2). By manually or automatically reviewing 4,900 pieces of code, our study reveals that: (1) large language models lack awareness of scenario-relevant security risks, which leads to the generation of over 75% vulnerable code on the SecurityEval benchmark; (2) LLMs such as GPT-3.5 and GPT-4 are unable to precisely identify vulnerabilities in the code they generated; (3) GPT-3.5 and GPT-4 can achieve 33.2%~59.6% success rates in repairing the insecure code produced by the 4 LLMs, but they both perform poorly when repairing self-produced code, indicating self-repair "blind spots". To address the limitation of a single round of repair, we developed a lightweight tool that prompts LLMs to construct safer source code through an iterative repair procedure based on the insights gained from our study. Experiments show that assisted by semantic analysis engines, our tool significantly improves the success rates of repair to 65.9%~85.5%.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper aims to explore whether large language models (LLMs) can serve as end-to-end secure code generators. Specifically, the researchers focus on the following points: 1. **Ability to generate secure code**: Can LLMs ensure the security of the code while generating functionally correct code? 2. **Ability to identify vulnerabilities**: Can LLMs accurately identify security vulnerabilities in the code they generate? 3. **Ability to fix vulnerabilities**: Can LLMs effectively fix security vulnerabilities in the code they generate? ### Research Background With the rapid development of large language models (such as GPT-4), these models are playing an increasingly important role in software engineering. However, whether the code generated by these models is secure has become an urgent issue. Although LLMs can generate functionally correct code, the code they generate may contain security vulnerabilities because the training datasets include many examples of insecure code. Therefore, it is not advisable to directly deploy code generated by LLMs in security-sensitive scenarios. ### Research Methods 1. **Dataset**: The researchers used the SecurityEval dataset, which is specifically designed to evaluate the security of machine learning-generated code and covers 69 common CWEs (Common Weakness Enumerations). 2. **Subjects**: Four popular LLMs were selected for the study, including GPT-3.5, GPT-4, Code Llama, and CodeGeeX2. 3. **Research Steps**: - **RQ1**: Evaluate the performance of LLMs in generating Python code, especially in security-sensitive scenarios. - **RQ2**: Evaluate the ability of LLMs to identify vulnerabilities in the generated code. - **RQ3**: Evaluate the ability of LLMs to fix vulnerabilities in the generated code. - **RQ4**: Explore whether iterative strategies can improve the ability of LLMs to fix vulnerabilities. ### Main Findings 1. **Performance in generating secure code**: All four LLMs performed poorly in generating secure code, with over 75% of the generated code containing security vulnerabilities. 2. **Ability to identify vulnerabilities**: LLMs were unable to accurately identify vulnerabilities in the code they generated. 3. **Ability to fix vulnerabilities**: Although LLMs showed a certain success rate (33.2%~59.6%) in fixing code generated by other LLMs, they performed poorly in fixing their own generated code, exhibiting "blind spots." 4. **Effectiveness of iterative strategies**: By introducing iterative repair strategies and combining semantic analysis engines (such as CodeQL and Bandit), the success rate of LLMs in fixing vulnerabilities can be significantly improved, reaching 65.9%~85.5%. ### Conclusion Although LLMs perform well in generating functionally correct code, they still face significant challenges in generating secure code. The research results indicate that the code generated by current LLMs should not be directly used in security-sensitive scenarios. To improve the security of code generated by LLMs, the researchers recommend adopting iterative repair strategies and using semantic analysis tools for assistance.

How Well Do Large Language Models Serve as End-to-End Secure Code Producers?

Can Large Language Models Find And Fix Vulnerable Software?

Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities

How secure is AI-generated Code: A Large-Scale Comparison of Large Language Models

Can We Trust Large Language Models Generated Code? A Framework for In-Context Learning, Security Patterns, and Code Evaluations Across Diverse LLMs

Ocassionally Secure: A Comparative Analysis of Code Generation Assistants

Large Language Models for Secure Code Assessment: A Multi-Language Empirical Study

Is Your AI-Generated Code Really Safe? Evaluating Large Language Models on Secure Code Generation with CodeSecEval

An Insight into Security Code Review with LLMs: Capabilities, Obstacles and Influential Factors

Fine Tuning Large Language Model for Secure Code Generation

What You See Is Not Always What You Get: An Empirical Study of Code Comprehension by Large Language Models

An Exploratory Study on Fine-Tuning Large Language Models for Secure Code Generation

Large Language Models and Code Security: A Systematic Literature Review

A Deep Dive into Large Language Model Code Generation Mistakes: What and Why?

CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion

Large Language Models for Code: Security Hardening and Adversarial Testing

Beyond Static Tools: Evaluating Large Language Models for Cryptographic Misuse Detection

Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attacks

Enhancing Large Language Models for Secure Code Generation: A Dataset-driven Study on Vulnerability Mitigation

DeepCode AI Fix: Fixing Security Vulnerabilities with Large Language Models