AutoTest: Evolutionary Code Solution Selection with Test Cases

Zhihua Duan,Jialin Wang
2024-08-22
Abstract:With the development of code generation techniques, selecting the correct code solution from multiple candidate solutions has become a crucial task. This study proposes AutoTest, a novel technique that combines automated test case generation with code solution execution to optimize the selection process using an evolutionary genetic algorithm. Firstly, AutoTest utilizes large pre-trained language models such as codegen-16B, code-davinci-002, and incoder-6B to provide code solutions and their corresponding test cases. Then, by executing the code solutions and evaluating their performance on the test cases, a consensus set is formed. Fine-grained ranking is achieved through the selection, mutation, and crossover mechanisms based on the evolutionary genetic algorithm, with the adjustment of alpha and beta parameters. Finally, the best code solution is chosen. AutoTest demonstrates significant performance improvements on the HumanEval benchmark test. The HumanEval dataset consists of 164 programming problems, and AutoTest achieves approximately a 10% improvement over the baseline method in terms of pass@1 score.
Software Engineering,Artificial Intelligence
What problem does this paper attempt to address?
The paper aims to address the problem of selecting the correct code from multiple candidate code solutions. Specifically, despite significant advancements in code generation technology in recent years, efficiently and accurately selecting the best solution from numerous candidates remains a challenge. To tackle this issue, the paper proposes a new method called AutoTest, which combines automatic test case generation with code execution, optimizing the code solution selection process through an evolutionary genetic algorithm. The specific workflow of AutoTest is as follows: 1. **Code Generation and Test Case Generation**: Utilize large pre-trained language models (such as codegen-16B, code-davinci-002, and incoder-6B) to generate code solutions and their corresponding test cases for each programming problem. 2. **Execution and Evaluation**: Execute the generated code solutions and evaluate their performance on the generated test cases to form a consensus set. 3. **Application of Evolutionary Genetic Algorithm**: By adjusting alpha and beta parameters and applying selection, mutation, and crossover mechanisms, finely rank the code solutions to ultimately select the optimal solution. Experimental results show that AutoTest outperforms baseline methods in the HumanEval benchmark test, particularly improving the pass@1 score by approximately 10%, demonstrating its potential in enhancing code generation quality. Additionally, the paper discusses future directions for improvement, including refining the scoring mechanism and optimizing algorithm parameters.