Abstract:Source code summaries are important for program comprehension and maintenance. However, there are plenty of programs with missing, outdated, or mismatched summaries. Recently, deep learning techniques have been exploited to automatically generate summaries for given code snippets. To achieve a profound understanding of how far we are from solving this problem and provide suggestions to future research, in this paper, we conduct a systematic and in-depth analysis of 5 state-of-the-art neural code summarization models on 6 widely used BLEU variants, 4 pre-processing operations and their combinations, and 3 widely used datasets. The evaluation results show that some important factors have a great influence on the model evaluation, especially on the performance of models and the ranking among the models. However, these factors might be easily overlooked. Specifically, (1) the BLEU metric widely used in existing work of evaluating code summarization models has many variants. Ignoring the differences among these variants could greatly affect the validity of the claimed results. Furthermore, we conduct human evaluations and find that the metric BLEU-DC is most correlated to human perception; (2) code pre-processing choices can have a large (from -18\% to +25\%) impact on the summarization performance and should not be neglected. We also explore the aggregation of pre-processing combinations and boost the performance of models; (3) some important characteristics of datasets (corpus sizes, data splitting methods, and duplication ratios) have a significant impact on model evaluation. Based on the experimental results, we give actionable suggestions for evaluating code summarization and choosing the best method in different scenarios. We also build a shared code summarization toolbox to facilitate future research.

Summary plus plus : Summarizing Chinese News Articles with Attention

Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization

HITS-based attentional neural model for abstractive summarization

A Syntax-Augmented and Headline-Aware Neural Text Summarization Method

Abstractive Document Summarization via Neural Model with Joint Attention

Selective and Coverage Multi-head Attention for Abstractive Summarization

An Abstractive Summarizer Based on Improved Pointer-Generator Network

Incorporating word attention with convolutional neural networks for abstractive summarization

A Convolutional Attention Network for Extreme Summarization of Source Code

CNewSum: A Large-scale Chinese News Summarization Dataset with Human-annotated Adequacy and Deducibility Level

Neural Abstractive Summarization with Structural Attention

AttSum: Joint Learning of Focusing and Summarization with Neural Attention.

Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method

A Neural Attention Model for Abstractive Sentence Summarization

Abstractive Summarization Using Attentive Neural Techniques

Template Based Chinese News Event Summarization

On the Evaluation of Neural Code Summarization

End-to-End Segmentation-based News Summarization

Contrastive Attention Mechanism for Abstractive Sentence Summarization

Dual-Level Contrastive Learning for Improving Conciseness of Summarization