Multitask-based Evaluation of Open-Source LLM on Software Vulnerability

Xin Yin,Chao Ni,Shaohua Wang

2024-07-07

Abstract:This paper proposes a pipeline for quantitatively evaluating interactive Large Language Models (LLMs) using publicly available datasets. We carry out an extensive technical evaluation of LLMs using Big-Vul covering four different common software vulnerability tasks. This evaluation assesses the multi-tasking capabilities of LLMs based on this dataset. We find that the existing state-of-the-art approaches and pre-trained Language Models (LMs) are generally superior to LLMs in software vulnerability detection. However, in software vulnerability assessment and location, certain LLMs (e.g., CodeLlama and WizardCoder) have demonstrated superior performance compared to pre-trained LMs, and providing more contextual information can enhance the vulnerability assessment capabilities of LLMs. Moreover, LLMs exhibit strong vulnerability description capabilities, but their tendency to produce excessive output significantly weakens their performance compared to pre-trained LMs. Overall, though LLMs perform well in some aspects, they still need improvement in understanding the subtle differences in code vulnerabilities and the ability to describe vulnerabilities to fully realize their potential. Our evaluation pipeline provides valuable insights into the capabilities of LLMs in handling software vulnerabilities.

Software Engineering

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

This paper aims to evaluate the performance of large language models (LLMs) in tasks related to software vulnerabilities. Specifically, the paper proposes an evaluation pipeline that quantitatively assesses LLMs using publicly available datasets (such as Big-Vul), covering four different common software vulnerability tasks:

1. **Vulnerability Detection**: Identifying whether there are vulnerabilities in the code.
2. **Vulnerability Assessment**: Evaluating the severity and potential impact of detected vulnerabilities.
3. **Vulnerability Location**: Precisely locating the specific position of vulnerabilities in the code or architecture.
4. **Vulnerability Description**: Providing a detailed explanation of identified vulnerabilities, including their origin, characteristics, and potential attack methods.

The paper seeks to comprehensively understand the multitasking capabilities of LLMs in these tasks through the following research questions (RQs):

- **RQ-1**: How do LLMs perform in vulnerability detection?
- **RQ-2**: How do LLMs perform in vulnerability assessment?
- **RQ-3**: How do LLMs perform in vulnerability location?
- **RQ-4**: How do LLMs perform in vulnerability description?

Through these research questions, the paper aims to systematically evaluate and compare the performance of LLMs with existing state-of-the-art methods and pre-trained language models (LMs) in software vulnerability management, thereby providing valuable insights for software quality assurance.