Abstract:Code Pre-trained Models (CodePTMs) based vulnerability detection have achieved promising results over recent years. However, these models struggle to generalize as they typically learn superficial mapping from source code to labels instead of understanding the root causes of code vulnerabilities, resulting in poor performance in real-world scenarios beyond the training instances. To tackle this challenge, we introduce VulLLM, a novel framework that integrates multi-task learning with Large Language Models (LLMs) to effectively mine deep-seated vulnerability features. Specifically, we construct two auxiliary tasks beyond the vulnerability detection task. First, we utilize the vulnerability patches to construct a vulnerability localization task. Second, based on the vulnerability features extracted from patches, we leverage GPT-4 to construct a vulnerability interpretation task. VulLLM innovatively augments vulnerability classification by leveraging generative LLMs to understand complex vulnerability patterns, thus compelling the model to capture the root causes of vulnerabilities rather than overfitting to spurious features of a single task. The experiments conducted on six large datasets demonstrate that VulLLM surpasses seven state-of-the-art models in terms of effectiveness, generalization, and robustness.

What problem does this paper attempt to address?

The paper attempts to address the issue of insufficient generalization ability of existing code pre-training models (CodePTMs) in detecting code vulnerabilities. Specifically, these models usually learn the superficial mapping relationship between source code and labels, rather than understanding the root causes of code vulnerabilities, resulting in poor performance when dealing with real-world scenarios beyond the training instances. To tackle this challenge, the paper proposes a new framework—VulLLM, which effectively mines deep vulnerability features through multi-task learning and large language models (LLMs). ### Main Issues: 1. **Insufficient Generalization Ability**: Existing code pre-training models perform poorly when handling out-of-distribution (OOD) data because they tend to capture superficial features rather than deep vulnerability features. 2. **Overfitting**: These models are prone to overfitting to spurious features of specific tasks, leading to poor generalization across different projects and programming styles. 3. **Adversarial Samples**: Existing methods struggle to distinguish adversarial samples such as simple identifier replacements, indicating that their predictions are influenced by irrelevant factors. ### Solution: - **VulLLM Framework**: Enhances the generalization ability and robustness of vulnerability detection through multi-task learning and large language models (LLMs). - **Auxiliary Tasks**: - **Vulnerability Localization Task**: Constructs a vulnerability localization task using vulnerability patches to identify vulnerable code elements. - **Vulnerability Explanation Task**: Utilizes GPT-4 to generate vulnerability explanations based on features extracted from patches. - **Multi-Task Instruction Fine-Tuning**: Adopts a multi-task learning paradigm to enable LLMs to learn deep vulnerability features rather than spurious features. - **Self-Verification Mechanism**: Introduces enhanced Chain-of-Thought (CoT) and Self-Verification (SV) mechanisms to avoid error accumulation and hallucinations, improving the reliability of LLMs. ### Experimental Results: - Experiments conducted on 6 large datasets show that VulLLM outperforms seven state-of-the-art models in terms of effectiveness, generalization ability, and robustness. - Compared to the best baseline model UniXcoder, VulLLM improves the F1 score by 8% and achieves an 8.58% improvement on four OOD datasets. - Under three types of adversarial attacks, VulLLM's overall F1 score improves by 68.08% compared to UniXcoder, demonstrating its stronger robustness. ### Contributions: - **Innovation**: Proposes using GPT-4's interpretability to generate vulnerability explanations, enhancing LLMs' understanding of vulnerabilities. - **Method**: Introduces the VulLLM framework, which uses LLMs for vulnerability detection through multi-task instruction fine-tuning. - **Evaluation**: Validates the method's effectiveness, generalization ability, and robustness across multiple datasets through extensive experiments, and open-sources the code and data.

Generalization-Enhanced Code Vulnerability Detection via Multi-Task Instruction Fine-Tuning

Function-Level Vulnerability Detection Through Fusing Multi-Modal Knowledge

Vul-LMGNNs: Fusing Language Models and Online-Distilled Graph Neural Networks for Code Vulnerability Detection

Towards Effectively Detecting and Explaining Vulnerabilities Using Large Language Models

Security Vulnerability Detection with Multitask Self-Instructed Fine-Tuning of Large Language Models

VulDetectBench: Evaluating the Deep Capability of Vulnerability Detection with Large Language Models

Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities

How Far Have We Gone in Vulnerability Detection Using Large Language Models

Large Language Model for Vulnerability Detection: Emerging Results and Future Directions

Outside the Comfort Zone: Analysing LLM Capabilities in Software Vulnerability Detection

Multitask-based Evaluation of Open-Source LLM on Software Vulnerability

Software Vulnerability Detection with GPT and In-Context Learning.

VTT-LLM: Advancing Vulnerability-to-Tactic-and-Technique Mapping through Fine-Tuning of Large Language Model

Investigating Large Language Models for Code Vulnerability Detection: An Experimental Study

An Empirical Study of Automated Vulnerability Localization with Large Language Models

Finetuning Large Language Models for Vulnerability Detection

StagedVulBERT: Multi-Granular Vulnerability Detection with a Novel Pre-trained Code Model

Large Language Models for Secure Code Assessment: A Multi-Language Empirical Study

M2CVD: Multi-Model Collaboration for Code Vulnerability Detection

Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG