Abstract:While large language models (LLMs) can already achieve strong performance on standard generic summarization benchmarks, their performance on more complex summarization task settings is less studied. Therefore, we benchmark LLMs on instruction controllable text summarization, where the model input consists of both a source article and a natural language requirement for desired summary characteristics. To this end, we curate an evaluation-only dataset for this task setting and conduct human evaluations of five LLM-based systems to assess their instruction-following capabilities in controllable summarization. We then benchmark LLM-based automatic evaluation for this task with 4 different evaluation protocols and 11 LLMs, resulting in 40 evaluation methods. Our study reveals that instruction controllable text summarization remains a challenging task for LLMs, since (1) all LLMs evaluated still make factual and other types of errors in their summaries; (2) no LLM-based evaluation methods can achieve a strong alignment with human annotators when judging the quality of candidate summaries; (3) different LLMs show large performance gaps in summary generation and evaluation capabilities. We make our collected benchmark InstruSum publicly available to facilitate future research in this direction.

What problem does this paper attempt to address?

The paper primarily focuses on the evaluation of large language models (LLMs) in the task of instruction-controllable text summarization and the research on automatic evaluation methods. Specifically, the paper attempts to address the following key issues: 1. **Definition and Challenges of Instruction-Controllable Text Summarization Task**: The paper first defines a new summarization task—instruction-controllable text summarization (referred to as "ins-controllable"), where the model not only needs to generate a summary based on the original document but also follow specific natural language instructions to adjust the content or format of the summary. This task setting aims to improve the flexibility and controllability of summaries to better meet the actual needs of users. 2. **Building Evaluation Benchmarks**: To evaluate the performance of LLMs under this task setting, the authors created a manually annotated dataset named INSTRU SUM, which includes articles selected from the XL-Sum dataset and their corresponding summary requirements. These summary requirements simulate different information needs and format preferences that users might have. Additionally, the authors collected human evaluation results for summaries generated by multiple LLMs, covering various dimensions such as overall quality, missing information, irrelevant information, and factual consistency. 3. **Benchmarking LLM Automatic Evaluation Methods**: Given that human evaluation is time-consuming and costly, the paper further investigates LLM-based automatic evaluation methods and explores the performance of different evaluation protocols (such as direct scoring, pairwise comparison, etc.) when combined with different LLM models. The study finds that although some methods can better evaluate the relative performance between summarization systems, there are still significant limitations in assessing the quality of individual summaries. In summary, the main goal of this paper is to gain a deeper understanding of the capabilities and limitations of LLMs in the task of instruction-controllable text summarization by defining a new task, building corresponding evaluation benchmarks, and testing automatic evaluation methods.

Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization

Benchmarking Large Language Models for News Summarization

On Learning to Summarize with Large Language Models as References

Controllable Multi-document Summarization: Coverage & Coherence Intuitive Policy with Large Language Model Based Rewards

UniSumEval: Towards Unified, Fine-Grained, Multi-Dimensional Summarization Evaluation for LLMs

In-context Learning of Large Language Models for Controlled Dialogue Summarization: A Holistic Benchmark and Empirical Analysis

Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization

Learning to Summarize from LLM-generated Feedback

CCSBench: Evaluating Compositional Controllability in LLMs for Scientific Document Summarization

Source Code Summarization in the Era of Large Language Models

UserSumBench: A Benchmark Framework for Evaluating User Summarization Approaches

Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation

Towards a Robust Retrieval-Based Summarization System

V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning

Evaluating Large Language Models at Evaluating Instruction Following

Text Summarization Using Large Language Models: A Comparative Study of MPT-7b-instruct, Falcon-7b-instruct, and OpenAI Chat-GPT Models

Can Large Language Models Serve as Evaluators for Code Summarization?

On Context Utilization in Summarization with Large Language Models

Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals?

Evaluate Summarization in Fine-Granularity: Auto Evaluation with LLM

An End-to-End Speech Summarization Using Large Language Model