Abstract:Unit testing aims to validate the correctness of software system units and has become an essential practice in software development and maintenance. However, it is incredibly time-consuming and labor-intensive for testing experts to write unit test cases manually, including test inputs (i.e., prefixes) and test oracles (i.e., assertions). Very recently, some techniques have been proposed to apply Large Language Models (LLMs) to generate unit assertions and have proven the potential in reducing manual testing efforts. However, there has been no systematic comparison of the effectiveness of these LLMs, and their pros and cons remain unexplored. To bridge this gap, we perform the first extensive study on applying various LLMs to automated assertion generation. The experimental results on two independent datasets show that studied LLMs outperform six state-of-the-art techniques with a prediction accuracy of 51.82% \(\sim\) 58.71% and 38.72% \(\sim\) 48.19%. The improvements achieve 29.60% and 12.47% on average. Besides, as a representative LLM, CodeT5 consistently outperforms all studied LLMs and all baselines on both datasets, with an average improvement of 13.85% and 26.64%, respectively. We also explore the performance of generated assertions in detecting real-world bugs, and find LLMs are able to detect 32 bugs from Defects4J on average, with an improvement of 52.38% against the most recent approach EditAS. Inspired by the findings, we construct a simplistic retrieval-and-repair-enhanced LLM-based approach by transforming the assertion generation problem into a program repair task for retrieved similar assertions. Surprisingly, such a simplistic approach can further improve the prediction accuracy of LLMs by 9.40% on average, leading to new records on both datasets. Besides, we provide additional discussions from different aspects (e.g., the impact of assertion types and test lengths) to illustrate the capacity and limitations of LLM-based approaches. Finally, we further pinpoint various practical guidelines (e.g., the improvement of multiple candidate assertions) for advanced LLM-based assertion generation in the near future. Overall, our work underscores the promising future of adopting off-the-shelf LLMs to generate accurate and meaningful assertions in real-world test cases and reduce the manual efforts of unit testing experts in practical scenarios.

$\forall$uto$\exists$val: Autonomous Assessment of LLMs in Formal Synthesis and Interpretation Tasks

$\forall$uto$\exists$$\lor\!\land$L: Autonomous Evaluation of LLMs for Truth Maintenance and Reasoning Tasks

Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and Credibility

Exploring Automated Assertion Generation Via Large Language Models

LLM4VV: Exploring LLM-as-a-Judge for Validation and Verification Testsuites

The Ability of Large Language Models to Evaluate Constraint-satisfaction in Agent Responses to Open-ended Requests

LLM4VV: Developing LLM-driven testsuite for compiler validation

Automated Assessment of Students' Code Comprehension using LLMs

Unveiling Context-Aware Criteria in Self-Assessing LLMs

LAAG-RV: LLM Assisted Assertion Generation for RTL Design Verification

LLMAuditor: A Framework for Auditing Large Language Models Using Human-in-the-Loop

On the Effectiveness of LLMs for Manual Test Verifications

Automata-based constraints for language model decoding

Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences

Can LLMs Reason in the Wild with Programs?

Can LLMs Replace Manual Annotation of Software Engineering Artifacts?

ALLURE: Auditing and Improving LLM-based Evaluation of Text using Iterative In-Context-Learning

Autoformalizing and Simulating Game-Theoretic Scenarios using LLM-augmented Agents

LLMs Could Autonomously Learn Without External Supervision

Finding Blind Spots in Evaluator LLMs with Interpretable Checklists

From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge