Abstract:Unit testing plays an essential role in detecting bugs in functionally-discrete program units (e.g., methods). Manually writing high-quality unit tests is time-consuming and laborious. Although the traditional techniques are able to generate tests with reasonable coverage, they are shown to exhibit low readability and still cannot be directly adopted by developers in practice. Recent work has shown the large potential of large language models (LLMs) in unit test generation. By being pre-trained on a massive developer-written code corpus, the models are capable of generating more human-like and meaningful test code. In this work, we perform the first empirical study to evaluate the capability of ChatGPT (i.e., one of the most representative LLMs with outstanding performance in code generation and comprehension) in unit test generation. In particular, we conduct both a quantitative analysis and a user study to systematically investigate the quality of its generated tests in terms of correctness, sufficiency, readability, and usability. We find that the tests generated by ChatGPT still suffer from correctness issues, including diverse compilation errors and execution failures (mostly caused by incorrect assertions); but the passing tests generated by ChatGPT almost resemble manually-written tests by achieving comparable coverage, readability, and even sometimes developers' preference. Our findings indicate that generating unit tests with ChatGPT could be very promising if the correctness of its generated tests could be further improved. Inspired by our findings above, we further propose ChatTester, a novel ChatGPT-based unit test generation approach, which leverages ChatGPT itself to improve the quality of its generated tests. ChatTester incorporates an initial test generator and an iterative test refiner. Our evaluation demonstrates the effectiveness of ChatTester by generating 34.3% more compilable tests and 18.7% more tests with correct assertions than the default ChatGPT. In addition to ChatGPT, we further investigate the generalization capabilities of ChatTester by applying it to two recent open-source LLMs (i.e., CodeLLama-Instruct and CodeFuse) and our results show that ChatTester can also improve the quality of tests generated by these LLMs.

Commit Message Generation Via ChatGPT: How Far Are We?

Neural-machine-translation-based Commit Message Generation: How Far Are We?

Commit Messages in the Age of Large Language Models

Automated Commit Message Generation with Large Language Models: An Empirical Study and Beyond

Using Large Language Models for Commit Message Generation: A Preliminary Study

RAG-Enhanced Commit Message Generation

A large-scale empirical study of commit message generation: models, datasets and evaluation

On the Evaluation of Commit Message Generation Models: An Experimental Study

Revisiting Learning-based Commit Message Generation.

From Commit Message Generation to History-Aware Commit Message Completion

ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark

Evaluating and Improving ChatGPT for Unit Test Generation

CoreGen: Contextualized Code Representation Learning for Commit Message Generation

Automatic Code Summarization via ChatGPT: How Far Are We?

A Closer Look at Different Difficulty Levels Code Generation Abilities of ChatGPT.

CommitBERT: Commit Message Generation Using Pre-Trained Programming Language Model

What Makes a Good Commit Message?

Automatically Generating Commit Messages from Diffs using Neural Machine Translation

Is It Hard to Generate Holistic Commit Message?

RACE: Retrieval-Augmented Commit Message Generation

ATOM: Commit Message Generation Based on Abstract Syntax Tree and Hybrid Ranking.