Abstract:Commit messages summarize code changes and help developers understand the intention. To alleviate human efforts in writing commit messages, researchers have proposed various automated commit message generation techniques, among which learning-based techniques have achieved great success in recent years. However, existing evaluation on learning-based commit message generation relies on the automatic metrics (e.g., BLEU) widely used in natural language processing (NLP) tasks, which are aggregated scores calculated based on the similarity between generated commit messages and the ground truth. Therefore, it remains unclear what generated commit messages look like and what kind of commit messages could be precisely generated by existing learning-based techniques. To fill this knowledge gap, this work performs the first study to systematically investigate the detailed commit messages generated by learning-based techniques. In particular, we first investigate the frequent patterns of the commit messages generated by state-of-the-art learning-based techniques. Surprisingly, we find the majority ( ~ 90%) of their generated commit messages belong to simple patterns (i.e., addition/removal/fix/avoidance patterns). To further explore the reasons, we then study the impact of datasets, input representations, and model components. We surprisingly find that existing learning-based techniques have competitive performance even when the inputs are only represented by change marks (i.e., "+"/"-"/""). It indicates that existing learning-based techniques poorly utilize syntax and semantics in the code while mostly focusing on change marks, which could be the major reason for generating so many pattern-matching commit messages. We also find that the pattern ratio in the training set might also positively affect the pattern ratio of generated commit messages; and model components might have different impact on the pattern ratio.

Using Large Language Models for Commit Message Generation: A Preliminary Study

Automated Commit Message Generation with Large Language Models: An Empirical Study and Beyond

Neural-machine-translation-based Commit Message Generation: How Far Are We?

Commit Messages in the Age of Large Language Models

A large-scale empirical study of commit message generation: models, datasets and evaluation

On the Evaluation of Commit Message Generation Models: An Experimental Study

Revisiting Learning-based Commit Message Generation.

Commit Message Generation Via ChatGPT: How Far Are We?

RAG-Enhanced Commit Message Generation

Context Conquers Parameters: Outperforming Proprietary LLM in Commit Message Generation

On the Evaluation of Large Language Models in Unit Test Generation

Can Large Language Models Be an Alternative to Human Evaluations?

A Survey on Evaluating Large Language Models in Code Generation Tasks

A Closer Look into Using Large Language Models for Automatic Evaluation

ATOM: Commit Message Generation Based on Abstract Syntax Tree and Hybrid Ranking.

Impact of Large Language Models on Generating Software Specifications

CommitBERT: Commit Message Generation Using Pre-Trained Programming Language Model

Leveraging Large Language Models for NLG Evaluation: Advances and Challenges

What Makes a Good Commit Message?

Is It Hard to Generate Holistic Commit Message?

A Comprehensive Analysis of the Effectiveness of Large Language Models As Automatic Dialogue Evaluators