Abstract:Commit messages are critical for code comprehension and software maintenance. Writing a high-quality message requires skill and effort. To support developers and reduce their effort on this task, several approaches have been proposed to automatically generate commit messages. Despite the promising performance reported, we have identified three significant and prevalent threats in these automated approaches: 1) the datasets used to train and evaluate these approaches contain a considerable amount of ‘noise’; 2) current approaches only consider commits of a limited diff size; and 3) current approaches can only generate the subject of a commit message, not the message body. The first limitation may let the models ‘learn’ inappropriate messages in the training stage, and also lead to inflated performance results in their evaluation. The other two threats can considerably weaken the practical usability of these approaches. Further, with the rapid emergence of large language models (LLMs) that show superior performance in many software engineering tasks, it is worth asking: can LLMs address the challenge of long diffs and whole message generation? This article first reports the results of an empirical study to assess the impact of these three threats on the performance of the state-of-the-art auto generators of commit messages. We collected commit data of the Top 1,000 most-starred Java projects in GitHub and systematically removed noisy commits with bot-submitted and meaningless messages. We then compared the performance of four approaches representative of the state-of-the-art before and after the removal of noisy messages, or with different lengths of commit diffs. We also conducted a qualitative survey with developers to investigate their perspectives on simply generating message subjects. Finally, we evaluate the performance of two representative LLMs, namely UniXcoder and ChatGPT, in generating more practical commit messages. The results demonstrate that generating commit messages is of great practical value, considerable work is needed to mature the current state-of-the-art, and LLMs can be an avenue worth trying to address the current limitations. Our analyses provide insights for future work to achieve better performance in practice.

Combining Code Context and Fine-grained Code Difference for Commit Message Generation

Neural-machine-translation-based Commit Message Generation: How Far Are We?

Commit Message Generation for Source Code Changes.

Context-aware Retrieval-based Deep Commit Message Generation

Learning Human-Written Commit Messages to Document Code Changes.

COME: Commit Message Generation with Modification Embedding

Automatically Generating Commit Messages from Diffs using Neural Machine Translation

CoreGen: Contextualized Code Representation Learning for Commit Message Generation

ESGen: Commit Message Generation Based on Edit Sequence of Code Change

A Sketch-Based Neural Model for Generating Commit Messages from Diffs

Revisiting Learning-based Commit Message Generation.

Generating commit messages from diffs using pointer-generator network

RAG-Enhanced Commit Message Generation

A large-scale empirical study of commit message generation: models, datasets and evaluation

Automated Commit Message Generation with Large Language Models: An Empirical Study and Beyond

An Empirical Study on Learning-based Techniques for Explicit and Implicit Commit Messages Generation

Retrieve-Guided Commit Message Generation with Semantic Similarity and Disparity

COLARE: Commit Classification Via Fine-grained Context-aware Representation of Code Changes

ATOM: Commit Message Generation Based on Abstract Syntax Tree and Hybrid Ranking.

Automatic Commit Message Generation: A Critical Review and Directions for Future Work

COMET: Generating Commit Messages using Delta Graph Context Representation