Abstract:Automated detection of software vulnerabilities is critical for enhancing security, yet existing methods often struggle with the complexity and diversity of modern codebases. In this paper, we introduce EnStack, a novel ensemble stacking framework that enhances vulnerability detection using natural language processing (NLP) techniques. Our approach synergizes multiple pre-trained large language models (LLMs) specialized in code understanding CodeBERT for semantic analysis, GraphCodeBERT for structural representation, and UniXcoder for cross-modal capabilities. By fine-tuning these models on the Draper VDISC dataset and integrating their outputs through meta-classifiers such as Logistic Regression, Support Vector Machines (SVM), Random Forest, and XGBoost, EnStack effectively captures intricate code patterns and vulnerabilities that individual models may overlook. The meta-classifiers consolidate the strengths of each LLM, resulting in a comprehensive model that excels in detecting subtle and complex vulnerabilities across diverse programming contexts. Experimental results demonstrate that EnStack significantly outperforms existing methods, achieving notable improvements in accuracy, precision, recall, and F1-score. This work highlights the potential of ensemble LLM approaches in code analysis tasks and offers valuable insights into applying NLP techniques for advancing automated vulnerability detection.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to improve the accuracy of software vulnerability detection in source code. Existing vulnerability detection methods often perform poorly when dealing with the complexity and diversity of modern codebases. The paper proposes a novel integrated stacking framework named EnStack, which enhances vulnerability detection capabilities by combining natural language processing (NLP) techniques. Specifically, EnStack integrates multiple pre - trained large - scale language models (LLMs), such as CodeBERT, GraphCodeBERT, and UniXcoder. These models are respectively good at semantic analysis of code, structural representation, and cross - modal capabilities. By fine - tuning these models on the Draper VDISC dataset and integrating their outputs using meta - classifiers (such as logistic regression, support vector machines, random forest, and XGBoost), EnStack can capture complex code patterns and vulnerabilities that a single model may miss. Experimental results show that EnStack significantly outperforms existing methods in terms of accuracy, precision, recall, and F1 - score, demonstrating the potential of the integrated LLM method in code analysis tasks. ### Main contributions of the paper: 1. **Proposed an integration - based stacking framework**: This framework combines multiple pre - trained large - scale language models (LLMs) and meta - classifiers to enhance vulnerability detection in source code. 2. **Comprehensively evaluated the EnStack framework**: Conducted extensive experiments on the Draper VDISC dataset, proving the superiority of EnStack in multiple performance indicators. 3. **Conducted ablation studies**: Analyzed the impact of different model combinations and meta - classifiers on detection performance, providing valuable insights into the effectiveness of the integration strategy. ### Specific description of the problem: - **Background**: With the rapid development of software development, the widespread existence of software vulnerabilities poses a serious security threat to individuals, organizations, and governments. Traditional vulnerability detection methods, such as manual code review and static analysis tools, have become difficult to cope with the complexity and scale of modern software systems. - **Challenges**: Existing methods based on large - scale language models (LLMs) usually focus on specific aspects of code representation. For example, CodeBERT focuses on semantic analysis, GraphCodeBERT emphasizes structural relationships, and UniXcoder attempts to unify cross - modal representation. These models may not be able to comprehensively capture the multifaceted nature of software vulnerabilities when used alone. - **Solution**: EnStack improves the overall performance of vulnerability detection by integrating multiple LLMs, taking advantage of their respective strengths, and combining meta - classifiers to optimize predictions. ### Method overview: - **Dataset**: Use the Draper VDISC dataset, which contains more than 1.27 million code functions with potential vulnerability labels. - **Model selection**: Select three pre - trained models, CodeBERT, GraphCodeBERT, and UniXcoder, and fine - tune them respectively for the semantic, structural, and cross - modal characteristics of the code. - **Integrated stacking**: Integrate the outputs of multiple models through meta - classifiers (such as logistic regression, support vector machines, random forest, and XGBoost) to form the final prediction result. - **Evaluation**: Comprehensively evaluate the model performance through indicators such as accuracy, precision, recall, F1 - score, and AUC - score. ### Experimental results: - **Individual model performance**: UniXcoder performs best among individual models, with an accuracy of 81.54% and an F1 - score of 81.49%. - **Stacking of a single LLM and a meta - classifier**: For example, after stacking UniXcoder with a support vector machine (SVM), the accuracy is increased to 81.36% and the F1 - score is increased to 81.89%. - **Integrated stacking of multiple LLMs**: After stacking GraphCodeBERT and UniXcoder (G + U) with SVM, the accuracy reaches 82.36% and the F1 - score reaches 82.28%, showing the best performance. ### Conclusion: EnStack significantly improves the accuracy of software vulnerability detection in source code by integrating multiple large - scale language models and meta - classifiers. This method not only outperforms existing individual models and traditional methods in performance but also provides new ideas for future code analysis tasks.

EnStack: An Ensemble Stacking Framework of Large Language Models for Enhanced Vulnerability Detection in Source Code

StagedVulBERT: Multi-Granular Vulnerability Detection with a Novel Pre-trained Code Model

DCDetector: An IoT terminal vulnerability mining system based on distributed deep ensemble learning under source code representation

Towards Effectively Detecting and Explaining Vulnerabilities Using Large Language Models

How Far Have We Gone in Vulnerability Detection Using Large Language Models

Harnessing Large Language Models for Software Vulnerability Detection: A Comprehensive Benchmarking Study

Finetuning Large Language Models for Vulnerability Detection

Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities

An extensive study of the effects of different deep learning models on code vulnerability detection in Python code

Large Language Models for Secure Code Assessment: A Multi-Language Empirical Study

Vulnerable JavaScript functions detection using stacking of convolutional neural networks

GRACE: Empowering LLM-based software vulnerability detection with graph structure and in-context learning

Detecting software vulnerabilities using Language Models

An Empirical Study of Automated Vulnerability Localization with Large Language Models

Prepare for Trouble and Make it Double. Supervised and Unsupervised Stacking for AnomalyBased Intrusion Detection

Dynamic stacking ensemble for cross-language code smell detection

Authorship Identification of Source Code Segments Written by Multiple Authors Using Stacking Ensemble Method

A tree-based stacking ensemble technique with feature selection for network intrusion detection

DLAP: A Deep Learning Augmented Large Language Model Prompting Framework for Software Vulnerability Detection

SAFE: Advancing Large Language Models in Leveraging Semantic and Syntactic Relationships for Software Vulnerability Detection