Abstract:Unit testing aims to validate the correctness of software system units and has become an essential practice in software development and maintenance. However, it is incredibly time-consuming and labor-intensive for testing experts to write unit test cases manually, including test inputs (i.e., prefixes) and test oracles (i.e., assertions). Very recently, some techniques have been proposed to apply Large Language Models (LLMs) to generate unit assertions and have proven the potential in reducing manual testing efforts. However, there has been no systematic comparison of the effectiveness of these LLMs, and their pros and cons remain unexplored. To bridge this gap, we perform the first extensive study on applying various LLMs to automated assertion generation. The experimental results on two independent datasets show that studied LLMs outperform six state-of-the-art techniques with a prediction accuracy of 51.82% \(\sim\) 58.71% and 38.72% \(\sim\) 48.19%. The improvements achieve 29.60% and 12.47% on average. Besides, as a representative LLM, CodeT5 consistently outperforms all studied LLMs and all baselines on both datasets, with an average improvement of 13.85% and 26.64%, respectively. We also explore the performance of generated assertions in detecting real-world bugs, and find LLMs are able to detect 32 bugs from Defects4J on average, with an improvement of 52.38% against the most recent approach EditAS. Inspired by the findings, we construct a simplistic retrieval-and-repair-enhanced LLM-based approach by transforming the assertion generation problem into a program repair task for retrieved similar assertions. Surprisingly, such a simplistic approach can further improve the prediction accuracy of LLMs by 9.40% on average, leading to new records on both datasets. Besides, we provide additional discussions from different aspects (e.g., the impact of assertion types and test lengths) to illustrate the capacity and limitations of LLM-based approaches. Finally, we further pinpoint various practical guidelines (e.g., the improvement of multiple candidate assertions) for advanced LLM-based assertion generation in the near future. Overall, our work underscores the promising future of adopting off-the-shelf LLMs to generate accurate and meaningful assertions in real-world test cases and reduce the manual efforts of unit testing experts in practical scenarios.

Utilizing Precise and Complete Code Context to Guide LLM in Automatic False Positive Mitigation

Automatically Inspecting Thousands of Static Bug Warnings with Large Language Model: How Far Are We?

An Insight into Security Code Review with LLMs: Capabilities, Obstacles and Influential Factors

LLMSA: A Compositional Neuro-Symbolic Approach to Compilation-free and Customizable Static Analysis

Reducing False Positives of Static Bug Detectors Through Code Representation Learning

Exploring Automatic Cryptographic API Misuse Detection in the Era of LLMs

Mitigating False Positive Static Analysis Warnings: Progress, Challenges, and Opportunities

LLM-Assisted Static Analysis for Detecting Security Vulnerabilities

Frustrated with Code Quality Issues? LLMs can Help!

Software Vulnerability and Functionality Assessment using LLMs

A Preliminary Study on Using Large Language Models in Software Pentesting

LMs: Understanding Code Syntax and Semantics for Code Analysis

Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities

Outside the Comfort Zone: Analysing LLM Capabilities in Software Vulnerability Detection

Helping LLMs Improve Code Generation Using Feedback from Testing and Static Analysis

Security Attacks on LLM-based Code Completion Tools

Security Code Review by LLMs: A Deep Dive into Responses

Beyond Static Tools: Evaluating Large Language Models for Cryptographic Misuse Detection

Large Language Models for Code Analysis: Do LLMs Really Do Their Job?

Exploring Automated Assertion Generation Via Large Language Models