GRAFT: Graph Retrieval Augmented Fine Tuning for Multi-Hop Query Summarization

Sonya Jin,Natalia Kokoromyti,Sunny Yu

Abstract:Traditional retrieval-augmented generation (RAG) approaches struggle with multi-hop reasoning and global query-focused summarization tasks over large document corpora, which require summarizing broad themes and contexts and a holistic knowledge of documents. We propose GRAFT (Graph Retrieval Augmented Fine-Tuning), a novel approach that combines the strengths of the Retrieval Augmented Fine-Tuning (RAFT) methodology and the GraphRAG technique. GRAFT fine-tunes large language models (LLMs) on a simulated imperfect retrieval setting, training the model to identify relevant documents and ignore distractors in the provided context. The model is then coupled with graphRAG at inference. To investigate the effectiveness of the GRAFT methodology, we constructed a knowledge graph using 74 Wikipedia source documents and extracted communities within this graph. We then summarized these communities, leveraging local and global relationships between documents for retrieval, fine-tuned a Microsoft Phi-2 model using the RAFT approach on a subset of the HotPotQA dataset, and evaluated its performance on a custom set of multi-hop and global questions generated from Wikipedia articles published in 2024. Our experimental results demonstrate that GRAFT outperforms baseline models, including the Baseline RAG model, the RAFT model, and the Baseline GraphRAG model, across various evaluation metrics like BERT, BLEU, ROUGE-1, and Semantic Similarity. In particular, GRAFT achieves the highest scores on global questions, showcasing its effectiveness in query-focused summarization tasks that require understanding broad themes and contexts over large document corpora.

Computer Science