Abstract:Large Language Models (LLMs) have been well-researched in many long-context tasks. However, due to high annotation costs, high-quality long-context summary datasets for training or evaluation are scarce, limiting further research. In this work, we introduce CNNSum, a new multi-scale Chinese long-context novel summarization benchmark, including four subsets, length covering 16k to 128k, 695 samples in total, the annotations are human-driven. We evaluate commercial and open-source models on CNNSum and conduct a detailed analysis. Based on the observations, we further conduct fine-tuning exploration with short-context summary data. In our study: (1) GPT-4o underperformed, due to excessive subjective commentary. (2) Currently, long-context summarization mainly relies on memory ability, small LLMs with stable longer context lengths are the most cost-effective. Using long data concatenated from short-context summaries makes a significant improvement. (3) Prompt templates may cause a large performance gap but can be mitigated through fine-tuning. (4) Fine-tuned Chat or Instruction versions may harm the Base model and further fine-tuning cannot bridge performance gap. (5) while models with RoPE base scaling exhibit strong extrapolation potential, their performance may vary significantly when combined with other interpolation methods and need careful selection. (6) CNNSum provides more reliable and insightful evaluation results than other benchmarks. We release CNNSum to advance research in this field (<a class="link-external link-https" href="https://github.com/CxsGhost/CNNSum" rel="external noopener nofollow">this https URL</a>).

In-context Learning of Large Language Models for Controlled Dialogue Summarization: A Holistic Benchmark and Empirical Analysis

Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization

On Context Utilization in Summarization with Large Language Models

On Learning to Summarize with Large Language Models as References

Benchmarking Large Language Models for News Summarization

Exploring the Dialogue Comprehension Ability of Large Language Models

CNNSum: Exploring Long-Conext Summarization with Large Language Models in Chinese Novels

Controllable Multi-document Summarization: Coverage & Coherence Intuitive Policy with Large Language Model Based Rewards

CNNSum: Exploring Long-Context Summarization with Large Language Models in Chinese Novels

LLM aided semi-supervision for Extractive Dialog Summarization

Improving Abstractive Dialogue Summarization with Speaker-Aware Supervised Contrastive Learning.

DialogSum: A Real-Life Scenario Dialogue Summarization Dataset

Exploring the Factual Consistency in Dialogue Comprehension of Large Language Models

Effective and Efficient Conversation Retrieval for Dialogue State Tracking with Implicit Text Summaries

Supervised Knowledge Makes Large Language Models Better In-context Learners

Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models

Baichuan2-Sum: Instruction Finetune Baichuan2-7B Model for Dialogue Summarization

An End-to-End Speech Summarization Using Large Language Model

MSAMSum: Towards Benchmarking Multi-lingual Dialogue Summarization

Prompting and Fine-Tuning of Small LLMs for Length-Controllable Telephone Call Summarization

Factual Dialogue Summarization via Learning from Large Language Models