Abstract:Selecting the best code solution from multiple generated ones is an essential task in code generation, which can be achieved by using some reliable validators (e.g., developer-written test cases) for assistance. Since reliable test cases are not always available and can be expensive to build in practice, researchers propose to automatically generate test cases to assess code solutions. However, when both code solutions and test cases are plausible and not reliable, selecting the best solution becomes challenging. Although some heuristic strategies have been proposed to tackle this problem, they lack a strong theoretical guarantee and it is still an open question whether an optimal selection strategy exists. Our work contributes in two ways. First, we show that within a Bayesian framework, the optimal selection strategy can be defined based on the posterior probability of the observed passing states between solutions and tests. The problem of identifying the best solution is then framed as an integer programming problem. Second, we propose an efficient approach for approximating this optimal (yet uncomputable) strategy, where the approximation error is bounded by the correctness of prior knowledge. We then incorporate effective prior knowledge to tailor code generation tasks. Both theoretical and empirical studies confirm that existing heuristics are limited in selecting the best solutions with plausible test cases. Our proposed approximated optimal strategy B4 significantly surpasses existing heuristics in selecting code solutions generated by large language models (LLMs) with LLM-generated tests, achieving a relative performance improvement by up to 50% over the strongest heuristic and 246% over the random selection in the most challenging scenarios. Our code is publicly available at <a class="link-external link-https" href="https://github.com/ZJU-CTAG/B4" rel="external noopener nofollow">this https URL</a>.

A Logical Approach to Generating Test Plans

Towards Optimal Concolic Testing

Using model checking to generate test cases for class testing

Guided test generation for coverage criteria

A novel approach of test case generation for Boolean specification

Reducing Test Cases with Causality Partitions.

Improving Generation of Object-Oriented Test Suites by Avoiding Redundant Tests

Extenics-based Test Case Generation for UML Activity Diagram.

Combining Dynamic Symbolic Execution, Machine Learning and Search-Based Testing to Automatically Generate Test Cases for Classes

Improving Defect Detection Ability of Derived Test Cases Based on Mutated UML Activity Diagrams

A Degraded ILP Approach for Test Suite Reduction

A Systematic Approach to Automatically Derive Test Cases from Use Cases Specified in Restricted Natural Languages.

Feedback-Directed Test Case Generation Based on UML Activity Diagrams

Generating Plans that Predict Themselves

Inferring Robot Task Plans from Human Team Meetings: A Generative Modeling Approach with Logic-Based Prior

Toward Generating Sufficiently Valid Test Case Results: A Method for Systematically Assigning Test Cases to Test Bench Configurations in a Scenario-Based Test Approach for Automated Vehicles

An Approach to Generate Integration Test Cases Based on UML Collaboration Diagrams

B4: Towards Optimal Assessment of Plausible Code Solutions with Plausible Tests

Leveraging Conditional Statement to Generate Acceptance Tests Automatically Via Traceable Sequence Generation

Test Scenario Generation for Context-Oriented Programs

Automated test data generation for aspect-oriented programs