Abstract:The current winning recipe for automatic summarization is using proprietary large-scale language models (LLMs) such as ChatGPT as is, or imitation learning from them as teacher models. While increasingly ubiquitous dependence on such large-scale language models is convenient, there remains an important question of whether small-scale models could have achieved competitive results, if we were to seek an alternative learning method -- that allows for a more cost-efficient, controllable, yet powerful summarizer. We present InfoSumm, a novel framework to distill a powerful summarizer based on the information-theoretic objective for summarization, without relying on either the LLM's capability or human-written references. To achieve this, we first propose a novel formulation of the desiderata of summarization (saliency, faithfulness and brevity) through the lens of mutual information between the original document and the summary. Based on this formulation, we start off from Pythia-2.8B as the teacher model, which is not yet capable of summarization, then self-train the model to optimize for the information-centric measures of ideal summaries. Distilling from the improved teacher, we arrive at a compact but powerful summarizer with only 568M parameters that performs competitively against ChatGPT, without ever relying on ChatGPT's capabilities. Extensive analysis demonstrates that our approach outperforms in-domain supervised models in human evaluation, let alone state-of-the-art unsupervised methods, and wins over ChatGPT in controllable summarization.

UniSumEval: Towards Unified, Fine-Grained, Multi-Dimensional Summarization Evaluation for LLMs

FineSurE: Fine-grained Summarization Evaluation using LLMs

Learning to Summarize from LLM-generated Feedback

UserSumBench: A Benchmark Framework for Evaluating User Summarization Approaches

UniSumm and SummZoo: Unified Model and Diverse Benchmark for Few-Shot Summarization

Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization

SummScore: A Comprehensive Evaluation Metric for Summary Quality Based on Cross-Encoder

OpinSummEval: Revisiting Automated Evaluation for Opinion Summarization

Benchmarking Large Language Models for News Summarization

Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation

SummEval: Re-evaluating Summarization Evaluation

UniSumm: Unified Few-shot Summarization with Multi-Task Pre-Training and Prefix-Tuning

AugSumm: towards generalizable speech summarization using synthetic labels from large language model

A Comparative Study of Quality Evaluation Methods for Text Summarization

Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articles

TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization

Information-Theoretic Distillation for Reference-less Summarization

Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method

GUMSum: Multi-Genre Data and Evaluation for English Abstractive Summarization

Multisumm: Towards A Unified Model For Multi-Lingual Abstractive Summarization