Large Language Models for Test-Free Fault Localization

Aidan Z.H. Yang,Ruben Martins,Claire Le Goues,Vincent J. Hellendoorn

2023-10-03

Abstract:Fault Localization (FL) aims to automatically localize buggy lines of code, a key first step in many manual and automatic debugging tasks. Previous FL techniques assume the provision of input tests, and often require extensive program analysis, program instrumentation, or data preprocessing. Prior work on deep learning for APR struggles to learn from small datasets and produces limited results on real-world programs. Inspired by the ability of large language models (LLMs) of code to adapt to new tasks based on very few examples, we investigate the applicability of LLMs to line level fault localization. Specifically, we propose to overcome the left-to-right nature of LLMs by fine-tuning a small set of bidirectional adapter layers on top of the representations learned by LLMs to produce LLMAO, the first language model based fault localization approach that locates buggy lines of code without any test coverage information. We fine-tune LLMs with 350 million, 6 billion, and 16 billion parameters on small, manually curated corpora of buggy programs such as the Defects4J corpus. We observe that our technique achieves substantially more confidence in fault localization when built on the larger models, with bug localization performance scaling consistently with the LLM size. Our empirical evaluation shows that LLMAO improves the Top-1 results over the state-of-the-art machine learning fault localization (MLFL) baselines by 2.3%-54.4%, and Top-5 results by 14.4%-35.6%. LLMAO is also the first FL technique trained using a language model architecture that can detect security vulnerabilities down to the code line level.

Software Engineering,Machine Learning

What problem does this paper attempt to address?

The paper attempts to address the problem of automatically locating faulty code lines (Fault Localization, FL) during software development. Traditional fault localization techniques usually require input tests and often involve extensive program analysis, program instrumentation, or data preprocessing. These methods have limitations in practical applications, especially in the absence of high-quality test cases. Additionally, existing deep learning-based automatic program repair (APR) techniques perform poorly on small datasets and have limited effectiveness on real-world programs.

To address these issues, the paper proposes a new fault localization method based on large language models (LLMs) called LLMAO. The main contributions of LLMAO are:

1. **Fault Localization Without Test Cases**: LLMAO is the first method capable of locating faulty code lines by fine-tuning large language models without relying on any test coverage information. This allows it to work effectively even in scenarios lacking test cases.

2. **High-Performance Fault Detection**: By fine-tuning on pre-trained language models of different scales, LLMAO significantly improves performance on fault localization tasks as the model scale increases. Experimental results show that LLMAO improves Top-1 and Top-5 fault localization results by 2.3%-54.4% and 14.4%-35.6%, respectively, compared to existing state-of-the-art machine learning fault localization techniques.

3. **Security Vulnerability Detection**: LLMAO is also the first machine learning fault localization technique capable of detecting code line-level security vulnerabilities, which is of great significance in the security field.

4. **Cross-Language and Domain Applicability**: LLMAO can be applied to various programming languages (such as C, Java, Python) and different types of defects (such as functional defects and security vulnerabilities), and it requires only a small amount of training data to achieve effective fine-tuning.

In summary, the paper proposes a novel fault localization method that does not require test cases by leveraging the powerful representation capabilities of large language models, significantly improving the accuracy and efficiency of fault localization.