Abstract:Unit testing aims to validate the correctness of software system units and has become an essential practice in software development and maintenance. However, it is incredibly time-consuming and labor-intensive for testing experts to write unit test cases manually, including test inputs (i.e., prefixes) and test oracles (i.e., assertions). Very recently, some techniques have been proposed to apply Large Language Models (LLMs) to generate unit assertions and have proven the potential in reducing manual testing efforts. However, there has been no systematic comparison of the effectiveness of these LLMs, and their pros and cons remain unexplored. To bridge this gap, we perform the first extensive study on applying various LLMs to automated assertion generation. The experimental results on two independent datasets show that studied LLMs outperform six state-of-the-art techniques with a prediction accuracy of 51.82% \(\sim\) 58.71% and 38.72% \(\sim\) 48.19%. The improvements achieve 29.60% and 12.47% on average. Besides, as a representative LLM, CodeT5 consistently outperforms all studied LLMs and all baselines on both datasets, with an average improvement of 13.85% and 26.64%, respectively. We also explore the performance of generated assertions in detecting real-world bugs, and find LLMs are able to detect 32 bugs from Defects4J on average, with an improvement of 52.38% against the most recent approach EditAS. Inspired by the findings, we construct a simplistic retrieval-and-repair-enhanced LLM-based approach by transforming the assertion generation problem into a program repair task for retrieved similar assertions. Surprisingly, such a simplistic approach can further improve the prediction accuracy of LLMs by 9.40% on average, leading to new records on both datasets. Besides, we provide additional discussions from different aspects (e.g., the impact of assertion types and test lengths) to illustrate the capacity and limitations of LLM-based approaches. Finally, we further pinpoint various practical guidelines (e.g., the improvement of multiple candidate assertions) for advanced LLM-based assertion generation in the near future. Overall, our work underscores the promising future of adopting off-the-shelf LLMs to generate accurate and meaningful assertions in real-world test cases and reduce the manual efforts of unit testing experts in practical scenarios.

Proof Automation with Large Language Models

Towards Large Language Models as Copilots for Theorem Proving in Lean

Automated Theorem Provers Help Improve Large Language Model Reasoning

Leveraging Large Language Models for Automated Proof Synthesis in Rust

Baldur: Whole-Proof Generation and Repair with Large Language Models

DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Data

NaturalProver: Grounded Mathematical Proof Generation with Language Models

Don't Trust: Verify -- Grounding LLM Quantitative Reasoning with Autoformalization

ImProver: Agent-Based Automated Proof Optimization

LeanDojo: Theorem Proving with Retrieval-Augmented Language Models

PAL: Program-aided Language Models

TheoremLlama: Transforming General-Purpose LLMs into Lean4 Experts

Evaluating the Ability of Large Language Models to Generate Verifiable Specifications in VeriFast

Towards Large Language Model Aided Program Refinement

How Do Humans Write Code? Large Models Do It the Same Way Too

Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification

Exploring Automated Assertion Generation Via Large Language Models

Large Language Models as an Indirect Reasoner: Contrapositive and Contradiction for Automated Reasoning

Steamroller Problems: An Evaluation of LLM Reasoning Capability with Automated Theorem Prover Strategies

Fixing Code Generation Errors for Large Language Models

LogicPro: Improving Complex Logical Reasoning via Program-Guided Learning