FlakyFix: Using Large Language Models for Predicting Flaky Test Fix Categories and Test Code Repair

Sakina Fatima,Hadi Hemmati,Lionel Briand

2024-08-31

Abstract:Flaky tests are problematic because they non-deterministically pass or fail for the same software version under test, causing confusion and wasting development effort. While machine learning models have been used to predict flakiness and its root causes, there is much less work on providing support to fix the problem. To address this gap, in this paper, we focus on predicting the type of fix that is required to remove flakiness and then repair the test code on that basis. We do this for a subset of flaky tests where the root cause of flakiness is in the test itself and not in the production code. One key idea is to guide the repair process with additional knowledge about the test's flakiness in the form of its predicted fix category. Thus, we first propose a framework that automatically generates labeled datasets for 13 fix categories and trains models to predict the fix category of a flaky test by analyzing the test code only. Our experimental results using code models and few-shot learning show that we can correctly predict most of the fix categories. To show the usefulness of such fix category labels for automatically repairing flakiness, we augment the prompts of GPT-3.5 Turbo, a Large Language Model (LLM), with such extra knowledge to request repair suggestions. The results show that our suggested fix category labels, complemented with in-context learning, significantly enhance the capability of GPT-3.5 Turbo in generating fixes for flaky tests. Based on the execution and analysis of a sample of GPT-repaired flaky tests, we estimate that a large percentage of such repairs (roughly between 51% and 83%) can be expected to pass. For the failing repaired tests, on average, 16% of the test code needs to be further changed for them to pass.

Software Engineering,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The paper primarily aims to address the issue of "flaky tests" in software development, which are test cases that pass or fail non-deterministically under the same software version. Specifically, the goals of the paper include:

1. **Defining Repair Categories**: First, the paper proposes a series of repair categories for flaky tests, which aim to provide practical guidance for developers to fix the tests.

2. **Predicting Repair Categories**: Given a flaky test case (without the need for production code), predict the most suitable repair category to apply. This step is achieved by analyzing the test code itself.

3. **Automatically Repairing Flaky Tests**: Integrate the predicted repair categories into an automated method for fixing flaky tests, using large language models (such as GPT 3.5 Turbo) to generate repair suggestions. The study found that the predicted repair category labels significantly improved the effectiveness of GPT in generating repair solutions.

Through the above three steps, the researchers not only defined multiple repair categories but also developed a set of heuristic rules and open-source scripts for automatically labeling flaky tests and their repair categories, and created a public dataset for training models. Additionally, they evaluated the performance of different models (including smaller pre-trained code models and large-scale language models) in predicting repair categories and generating repair code to demonstrate their effectiveness in practical applications.