Abstract:Retrieval-augmented generation (RAG) has revitalized Large Language Models (LLMs) by injecting non-parametric factual knowledge. Compared with long-context LLMs, RAG is considered an effective summarization tool in a more concise and lightweight manner, which can interact with LLMs multiple times using diverse queries to get comprehensive responses. However, the LLM-generated historical responses, which contain potentially insightful information, are largely neglected and discarded by existing approaches, leading to suboptimal results. In this paper, we propose \textit{graph of records} (\textbf{GoR}), which leverages historical responses generated by LLMs to enhance RAG for long-context global summarization. Inspired by the \textit{retrieve-then-generate} paradigm of RAG, we construct a graph by establishing an edge between the retrieved text chunks and the corresponding LLM-generated response. To further uncover the intricate correlations between them, GoR further features a \textit{graph neural network} and an elaborately designed \textit{BERTScore}-based objective for self-supervised model training, enabling seamless supervision signal backpropagation between reference summaries and node embeddings. We comprehensively compare GoR with 12 baselines across four long-context summarization datasets, and the results indicate that our proposed method reaches the best performance e.g., 15\%, 8\%, and 19\% improvement over retrievers w.r.t. Rouge-L, Rouge-1, and Rouge-2 on the WCEP dataset). Extensive experiments further demonstrate the effectiveness of GoR. Code is available at <a class="link-external link-https" href="https://github.com/ulab-uiuc/GoR" rel="external noopener nofollow">this https URL</a>

RGraph: Generating Reference Graphs for Better Machine Translation Evaluation

Multi-Reference Training with Pseudo-References for Neural Translation and Text Generation

MT-Ranker: Reference-free machine translation evaluation by inter-system ranking

Evaluating Optimal Reference Translations

Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying References

Quality and Quantity of Machine Translation References for Automatic Metrics

Simulated Multiple Reference Training Improves Low-Resource Machine Translation

On the Limitations of Reference-Free Evaluations of Generated Text

BLEU might be Guilty but References are not Innocent

RAGraph: A General Retrieval-Augmented Graph Learning Framework

Rephrasing the Reference for Non-Autoregressive Machine Translation

Reference Network for Neural Machine Translation

Not All Metrics Are Guilty: Improving NLG Evaluation with LLM Paraphrasing

One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation

Evaluation of Question Generation Needs More References

Graph-Based Collective Lexical Selection for Statistical Machine Translation.

Make Graph-based Referring Expression Comprehension Great Again through Expression-guided Dynamic Gating and Regression

Finding Replicable Human Evaluations via Stable Ranking Probability

Rethinking Round-Trip Translation for Machine Translation Evaluation

Graph of Records: Boosting Retrieval Augmented Generation for Long-context Summarization with Graphs

Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation