Mokav: Execution-driven Differential Testing with LLMs

Khashayar Etemadi,Bardia Mohammadi,Zhendong Su,Martin Monperrus

2024-06-15

Abstract:It is essential to detect functional differences in various software engineering tasks, such as automated program repair, mutation testing, and code refactoring. The problem of detecting functional differences between two programs can be reduced to searching for a difference exposing test (DET): a test input that results in different outputs on the subject programs. In this paper, we propose Mokav, a novel execution-driven tool that leverages LLMs to generate DETs. Mokav takes two versions of a program (P and Q) and an example test input. When successful, Mokav generates a valid DET, a test input that leads to different outputs on P and Q. Mokav iteratively prompts an LLM with a specialized prompt to generate new test inputs. At each iteration, Mokav provides execution-based feedback regarding previously generated tests until the LLM produces a DET. We evaluate Mokav on 1,535 pairs of Python programs collected from the Codeforces competition platform and 32 pairs of programs from the QuixBugs dataset. Our experiments show that Mokav outperforms the state-of-the-art, Pynguin and Differential Prompting, by a large margin. Mokav can generate DETs for 81.7% (1,255/1,535) of the program pairs in our benchmark (versus 4.9% for Pynguin and 37.3% for Differential Prompting). We demonstrate that all components in our system, including the iterative and execution-driven approaches, contribute to its high effectiveness.

Software Engineering

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in software engineering tasks, how to effectively detect the functional differences between two programs. Specifically, the authors propose a new tool, Mokav, which uses large - language models (LLMs) to generate test cases (Difference Exposing Test, DET) that can expose the functional differences between two programs. This problem is crucial in tasks such as automated program repair, mutation testing, and code refactoring.

### Specific description of the problem

1. **Background**:
- During the software development process, developers often need to handle multiple versions of a program and need to determine the functional differences or equivalences of these versions. For example, evaluating equivalence after refactoring, or ensuring the functional differences of mutants in mutation testing.
- Detecting the functional differences between two programs usually requires finding a Difference Exposing Test (DET), that is, an input that makes the two programs produce different outputs.

2. **Challenges**:
- Generating DET is a challenging task because it is necessary to explore the vast input space of the program and find those rare inputs that can trigger functional differences.
- Existing techniques such as symbolic execution, type - aware mutation, and genetic - algorithm - based code coverage optimization, although helpful in narrowing the search range, do not fully utilize the latest large - language models to understand program semantics and guide the search for DET.

3. **Solutions**:
- The paper proposes Mokav, a new tool based on LLMs, for generating DET. Mokav guides the LLM to generate test cases that can expose functional differences by iteratively providing execution feedback to the LLM.
- The workflow of Mokav includes an initial stage and an iterative stage:
- **Initial stage**: Mokav generates an initial prompt according to the given two program versions P and Q, an example test input, and execution data, and asks the LLM to generate the first candidate DET.
- **Iterative stage**: If the initial prompt fails to generate a valid DET, Mokav will continue to iteratively provide feedback to the LLM until a valid DET is generated or the maximum number of iterations is reached.

### Experimental results

- Mokav was evaluated on two datasets: QuixBugs and C4DET.
- The results show that Mokav successfully generated 81.7% of DETs on the C4DET dataset, significantly outperforming the two existing baseline methods, Pynguin (4.9%) and Differential Prompting (37.3%).

### Summary

The main contribution of the paper is to propose a new difference - testing method based on LLMs, Mokav, which efficiently generates difference - exposing test cases by iteratively providing execution feedback, thereby significantly improving the ability to detect program functional differences.