Graph of Records: Boosting Retrieval Augmented Generation for Long-context Summarization with Graphs

Haozhen Zhang,Tao Feng,Jiaxuan You

2024-10-15

Abstract:Retrieval-augmented generation (RAG) has revitalized Large Language Models (LLMs) by injecting non-parametric factual knowledge. Compared with long-context LLMs, RAG is considered an effective summarization tool in a more concise and lightweight manner, which can interact with LLMs multiple times using diverse queries to get comprehensive responses. However, the LLM-generated historical responses, which contain potentially insightful information, are largely neglected and discarded by existing approaches, leading to suboptimal results. In this paper, we propose \textit{graph of records} (\textbf{GoR}), which leverages historical responses generated by LLMs to enhance RAG for long-context global summarization. Inspired by the \textit{retrieve-then-generate} paradigm of RAG, we construct a graph by establishing an edge between the retrieved text chunks and the corresponding LLM-generated response. To further uncover the intricate correlations between them, GoR further features a \textit{graph neural network} and an elaborately designed \textit{BERTScore}-based objective for self-supervised model training, enabling seamless supervision signal backpropagation between reference summaries and node embeddings. We comprehensively compare GoR with 12 baselines across four long-context summarization datasets, and the results indicate that our proposed method reaches the best performance e.g., 15\%, 8\%, and 19\% improvement over retrievers w.r.t. Rouge-L, Rouge-1, and Rouge-2 on the WCEP dataset). Extensive experiments further demonstrate the effectiveness of GoR. Code is available at <a class="link-external link-https" href="https://github.com/ulab-uiuc/GoR" rel="external noopener nofollow">this https URL</a>

Computation and Language,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is that existing Retrieval - Augmented Generation (RAG) methods fail to fully utilize the historical responses generated by large - language models (LLM) when handling the global summarization task of long - text. Specifically, although these historical responses contain valuable information, current methods often overlook them, resulting in sub - optimal results. To overcome this challenge, the paper proposes the "Graph of Records (GoR)" method, which enhances the performance of RAG in long - text global summarization by constructing a graph structure that connects retrieved text fragments with responses generated by LLM.

### Main contributions:
1. **Proposing "Graph of Records (GoR)"**: Utilize and organize the historical responses generated by LLM as a record graph to strengthen the performance of RAG in long - text global summarization. The research reveals that the fine - grained associations between LLM historical responses and text fragments in long documents can be effectively mined and utilized, thereby improving the performance of RAG.
2. **Utilizing Graph Neural Networks (GNN) and BERTScore - based objective functions**: Design a self - supervised training objective, optimize node embeddings through graph neural networks, enabling them to adaptively reflect the semantics and complex associations of input queries. In addition, the indirect supervision signals from self - supervised labels are crucial for the effective optimization of node embeddings.
3. **Experimental verification**: Evaluate the proposed method on four long - text summarization datasets. The experimental results show that GoR significantly outperforms multiple competing baselines. For example, on the WCEP dataset, GoR improves the Retrieval - Augmented Generation (RAG) by 15%, 8% and 19% on Rouge - L, Rouge - 1 and Rouge - 2 metrics respectively compared to the retriever.

### Specific problems solved:
- **Utilization of historical responses**: Existing methods ignore the historical responses generated by LLM, and the information contained in these responses may be very useful for summary generation. GoR fully utilizes this information by constructing a graph structure to associate these historical responses with retrieved text fragments.
- **Mining of complex associations**: There are complex associations between text fragments in long documents and responses generated by LLM, and these associations are difficult to capture by traditional semantic similarity methods. GoR effectively mines these associations through graph neural networks and self - supervised training objectives.
- **Lack of supervision signals**: Unlike local summarization tasks, global summarization tasks lack clear labels and it is difficult to directly back - propagate effective supervision signals. GoR indirectly provides supervision signals through the BERTScore - based ranking mechanism and optimizes node embeddings.

In conclusion, by proposing the "Graph of Records (GoR)" method, this paper solves the problem that existing RAG methods fail to fully utilize historical responses in long - text global summarization tasks, and significantly improves the quality and performance of summaries.