Abstract:To support software developers in understanding and maintaining programs, various automatic (source) code summarization techniques have been proposed to generate a concise natural language summary (i.e., comment) for a given code snippet. Recently, the emergence of large language models (LLMs) has led to a great boost in the performance of code-related tasks. In this paper, we undertake a systematic and comprehensive study on code summarization in the era of LLMs, which covers multiple aspects involved in the workflow of LLM-based code summarization. Specifically, we begin by examining prevalent automated evaluation methods for assessing the quality of summaries generated by LLMs and find that the results of the GPT-4 evaluation method are most closely aligned with human evaluation. Then, we explore the effectiveness of five prompting techniques (zero-shot, few-shot, chain-of-thought, critique, and expert) in adapting LLMs to code summarization tasks. Contrary to expectations, advanced prompting techniques may not outperform simple zero-shot prompting. Next, we investigate the impact of LLMs' model settings (including top\_p and temperature parameters) on the quality of generated summaries. We find the impact of the two parameters on summary quality varies by the base LLM and programming language, but their impacts are similar. Moreover, we canvass LLMs' abilities to summarize code snippets in distinct types of programming languages. The results reveal that LLMs perform suboptimally when summarizing code written in logic programming languages compared to other language types. Finally, we unexpectedly find that CodeLlama-Instruct with 7B parameters can outperform advanced GPT-4 in generating summaries describing code implementation details and asserting code properties. We hope that our findings can provide a comprehensive understanding of code summarization in the era of LLMs.

LLMs as Evaluators: A Novel Approach to Evaluate Bug Report Summarization

Analysis on LLMs Performance for Code Summarization

Can Large Language Models Serve as Evaluators for Code Summarization?

A Comparative Study of Quality Evaluation Methods for Text Summarization

Evaluate Summarization in Fine-Granularity: Auto Evaluation with LLM

Can LLMs Replace Manual Annotation of Software Engineering Artifacts?

Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization

Towards Understanding the Robustness of LLM-based Evaluations under Perturbations

Benchmarking Large Language Models for News Summarization

Optimizing the role of human evaluation in LLM-based spoken document summarization systems

LaMSUM: Creating Extractive Summaries of User Generated Content using LLMs

What's Wrong? Refining Meeting Summaries with LLM Feedback

Source Code Summarization in the Era of Large Language Models

Analyzing LLM Behavior in Dialogue Summarization: Unveiling Circumstantial Hallucination Trends

On Learning to Summarize with Large Language Models as References

Applicability of Large Language Models and Generative Models for Legal Case Judgement Summarization

Using LLMs in Software Requirements Specifications: An Empirical Evaluation

Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summaries

Evaluating Large Language Models on Financial Report Summarization: An Empirical Study

Comparative Analysis of Open-Source Language Models in Summarizing Medical Text Data

Leveraging the Power of LLMs: A Fine-Tuning Approach for High-Quality Aspect-Based Summarization