Abstract:Commit Message Generation (CMG) approaches aim to automatically generate commit messages based on given code diffs, which facilitate collaboration among developers and play a critical role in Open-Source Software (OSS). Very recently, Large Language Models (LLMs) have demonstrated extensive applicability in diverse code-related task. But few studies systematically explored their effectiveness using LLMs. This paper conducts the first comprehensive experiment to investigate how far we have been in applying LLM to generate high-quality commit messages. Motivated by a pilot analysis, we first clean the most widely-used CMG dataset following practitioners' criteria. Afterward, we re-evaluate diverse state-of-the-art CMG approaches and make comparisons with LLMs, demonstrating the superior performance of LLMs against state-of-the-art CMG approaches. Then, we further propose four manual metrics following the practice of OSS, including Accuracy, Integrity, Applicability, and Readability, and assess various LLMs accordingly. Results reveal that GPT-3.5 performs best overall, but different LLMs carry different advantages. To further boost LLMs' performance in the CMG task, we propose an Efficient Retrieval-based In-Context Learning (ICL) framework, namely ERICommiter, which leverages a two-step filtering to accelerate the retrieval efficiency and introduces semantic/lexical-based retrieval algorithm to construct the ICL examples. Extensive experiments demonstrate the substantial performance improvement of ERICommiter on various LLMs for code diffs of different programming languages. Meanwhile, ERICommiter also significantly reduces the retrieval time while keeping almost the same performance. Our research contributes to the understanding of LLMs' capabilities in the CMG field and provides valuable insights for practitioners seeking to leverage these tools in their workflows.

An Extended GHKM Algorithm for Inducing Lambda-SCFG.

MKGL: Mastery of a Three-Word Language

KG-GPT: A General Framework for Reasoning on Knowledge Graphs Using Large Language Models

Macro Grammars and Holistic Triggering for Efficient Semantic Parsing

Semantic Construction Grammar: Bridging the NL / Logic Divide

Knowledge Solver: Teaching LLMs to Search for Domain Knowledge from Knowledge Graphs

An EM Algorithm for SCFG in Formal Syntax-Based Translation

Fusing topology contexts and logical rules in language models for knowledge graph completion

Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomena

Semantic Parsing with Candidate Expressions for Knowledge Base Question Answering

Automated Commit Message Generation with Large Language Models: An Empirical Study and Beyond

G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model

GrammarGPT: Exploring Open-Source LLMs for Native Chinese Grammatical Error Correction with Supervised Fine-Tuning

An Enhanced Prompt-Based LLM Reasoning Scheme via Knowledge Graph-Integrated Collaboration

Human-Like Code Quality Evaluation through LLM-based Recursive Semantic Comprehension

Large-scale CCG Induction from the Groningen Meaning Bank

TransLLaMa: LLM-based Simultaneous Translation System

Semantic Role Labeling for Learner Chinese: the Importance of Syntactic Parsing and L2-L1 Parallel Data

Parsing into Variable-in-situ Logico-Semantic Graphs.

Geo-FuB: A Method for Constructing an Operator-Function Knowledge Base for Geospatial Code Generation Tasks Using Large Language Models