Abstract:There is an increasing trend to mine vulnerabilities from software repositories and use machine learning techniques to automatically detect software vulnerabilities. A fundamental but unresolved research question is: how do different factors in the mining and learning process impact the accuracy of identifying vulnerabilities in software projects of varying characteristics? Substantial research has been dedicated in this area, including source code static analysis, software repository mining, and NLP-based machine learning. However, practitioners lack experience regarding the key factors for building a baseline model of the state-of-the-art. In addition, there lacks of experience regarding the transferability of the vulnerability signatures from project to project. This study investigates how the combination of different vulnerability features and three representative machine learning models impact the accuracy of vulnerability detection in 17 real-world projects. We examine two types of vulnerability representations: 1) code features extracted through NLP with varying tokenization strategies and three different embedding techniques (bag-of-words, word2vec, and fastText) and 2) a set of eight architectural metrics that capture the abstract design of the software systems. The three machine learning algorithms include a random forest model, a support vector machines model, and a residual neural network model. The analysis shows a recommended baseline model with signatures extracted through bag-of-words embedding, combined with the random forest, consistently increases the detection accuracy by about 4% compared to other combinations in all 17 projects. Furthermore, we observe the limitation of transferring vulnerability signatures across domains based on our experiments.

Network Vulnerability Analysis Using Text Mining

Categorizing and Predicting Invalid Vulnerabilities on Common Vulnerabilities and Exposures

Threat Assessment for Power Industrial Control System Based on Descriptive Vulnerability Text

Vulnerability Detection For Smart Grid Devices Via Static Analysis

Patchmatch: A Tool for Locating Patches of Open Source Project Vulnerabilities

Network computer security hidden dangers and vulnerability mining technology

A Mining Approach to Obtain the Software Vulnerability Characteristics

An empirical study of text-based machine learning models for vulnerability detection

A Mining Method for Computer Vulnerability Correlation

Combining Software Metrics and Text Features for Vulnerable File Prediction

Explaining the Contributing Factors for Vulnerability Detection in Machine Learning

Vulnerability Assessment on WAMS Communication System Based on Attack Tree Model

An Analysis of Network Protocol Vulnerability Mining Using Fuzz Testing Combined with Deep Learning Models

Mining Privilege Escalation Paths for Network Vulnerability Analysis

Analysis of Risk Factors of Coal Chemical Enterprises Based on Text Mining.

Predicting Vulnerable Components via Text Mining or Software Metrics? An Effort-Aware Perspective

A Survey of Automatic Exploitation of Binary Vulnerabilities

Research on automatic mining and utilization of vulnerability in power web system

Vulnerability Detection with Graph Attention Network and Metric Learning

Research progress of Web vulnerability mining based on machine learning