Abstract:Research on automated text summarization relies heavily on human and automatic evaluation. While recent work on human evaluation mainly adopted intrinsic evaluation methods, judging the generic quality of text summaries, e.g. informativeness and coherence, our work focuses on evaluating the usefulness of text summaries with extrinsic methods. We carefully design three different downstream tasks for extrinsic human evaluation of summaries, i.e., question answering, text classification and text similarity assessment. We carry out experiments using system rankings and user behavior data to evaluate the performance of different summarization models. We find summaries are particularly useful in tasks that rely on an overall judgment of the text, while being less effective for question answering tasks. The results show that summaries generated by fine-tuned models lead to higher consistency in usefulness across all three tasks, as rankings of fine-tuned summarization systems are close across downstream tasks according to the proposed extrinsic metrics. Summaries generated by models in the zero-shot setting, however, are found to be biased towards the text classification and similarity assessment tasks, due to its general and less detailed summary style. We further evaluate the correlation of 14 intrinsic automatic metrics with human criteria and show that intrinsic automatic metrics perform well in evaluating the usefulness of summaries in the question-answering task, but are less effective in the other two tasks. This highlights the limitations of relying solely on intrinsic automatic metrics in evaluating the performance and usefulness of summaries.

Better Metrics to Automatically Predict the Quality of a Text Summary

A Comparative Study of Quality Evaluation Methods for Text Summarization

A Statistical Analysis of Summarization Evaluation Metrics Using Resampling Methods

Re-evaluating Evaluation in Text Summarization

Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metrics

What Have We Achieved on Text Summarization?

Metrics Also Disagree in the Low Scoring Range: Revisiting Summarization Evaluation Metrics

Correction of Errors in Preference Ratings from Automated Metrics for Text Generation

Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation

RCSUM: To build a summarization system directly generating summaries with evaluation metrics

SummScore: A Comprehensive Evaluation Metric for Summary Quality Based on Cross-Encoder

Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics

A Critical Look at Meta-evaluating Summarisation Evaluation Metrics

Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning

DocAsRef: an Empirical Study on Repurposing Reference-based Summary Quality Metrics As Reference-free Metrics

Evaluating Code Summarization with Improved Correlation with Human Assessment.

An Anchor-Based Automatic Evaluation Metric for Document Summarization.

Understanding the Extent to which Summarization Evaluation Metrics Measure the Information Quality of Summaries

Towards objectively evaluating the quality of generated medical summaries

Is Summary Useful or Not? An Extrinsic Human Evaluation of Text Summaries on Downstream Tasks

OpinSummEval: Revisiting Automated Evaluation for Opinion Summarization