Quality Assurance for Automated Commit Message Generation.

Bei Wang,Meng Yan,Zhongxin Liu,Ling Xu,Xin Xia,Xiaohong Zhang,Dan Yang
DOI: https://doi.org/10.1109/saner50967.2021.00032
2021-01-01
Abstract:Many automated commit message generation (CMG) approaches have been proposed for facilitating the understanding of software changes. They are shown to be promising and can generate commit messages that are semantically relevant to the reference messages for a number of commits. However, a large proportion (over 50%) of semantically irrelevant commit messages are also generated simultaneously. Such messages may mislead developers, require additional efforts of developers to confirm and filter out, and hinder the application of existing CMG approaches in practice. For tackling this problem, prior work mainly focuses on proposing new methods to improve the generation accuracy. However, another promising way for bridging the gap between CMG approaches and the practice has not been well investigated, which is: can we automatically assure the semantic relevance of the generated messages?To that end, in this work, we propose an automated Quality A ssurance framework for commit message generation (QAcom). QAcom can assure the quality of generated commit messages by automatically filtering out the semantically-irrelevant generated messages and preserving the semantically-relevant ones as many as possible. In particular, QAcom consists of a Collaborative-Filtering-based (CF) component and a Retrieval-based (RE) component. Given a commit message generated by a CMG approach, QAcom estimates whether this generated message is semantically relevant to its ground truth, which is unknown when estimating, based on both the collaborative filtering algorithm and the similarity between this commit and historical commits. We evaluate the effectiveness of QAcom by "plugging" it in three state-of-the-art CMG approaches. Experimental results on three public datasets show that QAcom can effectively filter out semantically-irrelevant generated messages and preserve semantically-relevant ones.
What problem does this paper attempt to address?