Abstract:Security vulnerability repair is a difficult task that is in dire need of automation. Two groups of techniques have shown promise: (1) large code language models (LLMs) that have been pre-trained on source code for tasks such as code completion, and (2) automated program repair (APR) techniques that use deep learning (DL) models to automatically fix software bugs. This paper is the first to study and compare Java vulnerability repair capabilities of LLMs and DL-based APR models. The contributions include that we (1) apply and evaluate five LLMs (Codex, CodeGen, CodeT5, PLBART and InCoder), four fine-tuned LLMs, and four DL-based APR techniques on two real-world Java vulnerability benchmarks (Vul4J and VJBench), (2) design code transformations to address the training and test data overlapping threat to Codex, (3) create a new Java vulnerability repair benchmark VJBench, and its transformed version VJBench-trans and (4) evaluate LLMs and APR techniques on the transformed vulnerabilities in VJBench-trans. Our findings include that (1) existing LLMs and APR models fix very few Java vulnerabilities. Codex fixes 10.2 (20.4%), the most number of vulnerabilities. (2) Fine-tuning with general APR data improves LLMs' vulnerability-fixing capabilities. (3) Our new VJBench reveals that LLMs and APR models fail to fix many Common Weakness Enumeration (CWE) types, such as CWE-325 Missing cryptographic step and CWE-444 HTTP request smuggling. (4) Codex still fixes 8.3 transformed vulnerabilities, outperforming all the other LLMs and APR models on transformed vulnerabilities. The results call for innovations to enhance automated Java vulnerability repair such as creating larger vulnerability repair training data, tuning LLMs with such data, and applying code simplification transformation to facilitate vulnerability repair.

Using Sequence-to-Sequence Learning for Repairing C Vulnerabilities

SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair

CRepair: CVAE-based Automatic Vulnerability Repair Technology

Neural Transfer Learning for Repairing Security Vulnerabilities in C Code

Enabling Automatic Repair of Source Code Vulnerabilities Using Data-Driven Methods

How Effective Are Neural Networks for Fixing Security Vulnerabilities

Pre-trained Model-based Automated Software Vulnerability Repair: How Far are We?

Automated software vulnerability detection with machine learning

An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation

SySeVR: A Framework for Using Deep Learning to Detect Software Vulnerabilities

A Survey of Automatic Software Vulnerability Detection, Program Repair, and Defect Prediction Techniques

A Software-Repair Robot based on Continual Learning

Automated Software Vulnerability Patching using Large Language Models

NAVRepair: Node-type Aware C/C++ Code Vulnerability Repair

DeepVulSeeker: A novel vulnerability identification framework via code graph structure and pre-training mechanism

Code Vulnerability Detection Based on Deep Sequence and Graph Models: A Survey

Survey of Source Code Vulnerability Analysis Based on Deep Learning

LLM-Powered Code Vulnerability Repair with Reinforcement Learning and Semantic Reward

CSGVD: A deep learning approach combining sequence and graph embedding for source code vulnerability detection

Enhanced Automated Code Vulnerability Repair using Large Language Models

Transformer-based Vulnerability Detection in Code at EditTime: Zero-shot, Few-shot, or Fine-tuning?