Revisiting the Performance of Deep Learning-Based Vulnerability Detection on Realistic Datasets

Partha Chakraborty,Krishna Kanth Arumugam,Mahmoud Alfadel,Meiyappan Nagappan,Shane McIntosh

2024-07-03

Abstract:The impact of software vulnerabilities on everyday software systems is significant. Despite deep learning models being proposed for vulnerability detection, their reliability is questionable. Prior evaluations show high recall/F1 scores of up to 99%, but these models underperform in practical scenarios, particularly when assessed on entire codebases rather than just the fixing commit. This paper introduces Real-Vul, a comprehensive dataset representing real-world scenarios for evaluating vulnerability detection models. Evaluating DeepWukong, LineVul, ReVeal, and IVDetect shows a significant drop in performance, with precision decreasing by up to 95 percentage points and F1 scores by up to 91 points. Furthermore, Model performance fluctuates based on vulnerability characteristics, with better F1 scores for information leaks or code injection than for path resolution or predictable return values. The results highlight a significant performance gap that needs addressing before deploying deep learning-based vulnerability detection in practical settings. Overfitting is identified as a key issue, and an augmentation technique is proposed, potentially improving performance by up to 30%. Contributions include a dataset creation approach for better model evaluation, Real-Vul dataset, and empirical evidence of deep learning models struggling in real-world settings.

Software Engineering,Artificial Intelligence,Cryptography and Security,Machine Learning

What problem does this paper attempt to address?

The paper attempts to address the issue of poor performance of existing deep learning models in detecting software vulnerabilities in real-world scenarios. Although previous studies have reported recall/F1 scores as high as 99% on certain datasets, the performance of these models significantly drops when applied to entire codebases rather than just patched commits. The authors found that existing datasets (such as synthetic datasets and Oracle-based datasets) fail to accurately reflect the complexity and diversity of real-world vulnerabilities and suffer from inconsistent labeling. Therefore, the paper proposes a new dataset—Real-Vul, which aims to more realistically simulate the application scenarios of vulnerability detection models in real environments to evaluate their true performance.

Specifically, the main contributions of the paper include:
1. **Proposing the Real-Vul dataset**: This dataset includes complete codebases, which can more realistically reflect the situation of vulnerability detection models in actual applications.
2. **Re-evaluating existing models**: The paper re-evaluates four state-of-the-art vulnerability detection models (DeepWukong, LineVul, ReVeal, and IVDetect) using the Real-Vul dataset and finds that their performance significantly drops in real-world scenarios.
3. **Analyzing the reasons for performance decline**: Through visualization and manual analysis, the paper reveals issues such as model overfitting and difficulty in distinguishing between vulnerable and uncertain code, and proposes an enhancement technique that can improve model performance by up to 30%.
4. **Layered evaluation by vulnerability type and severity**: The paper also conducts a layered evaluation of model performance by vulnerability type and severity, finding that models perform better in detecting certain types of vulnerabilities (such as information leakage and code injection) but worse in detecting other types (such as path traversal and predictable return values).

In summary, the paper aims to reveal the limitations of existing deep learning models in vulnerability detection by proposing a dataset that better aligns with real-world application scenarios and provides directions for future improvements.