Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering

Zhentao Xu,Mark Jerome Cruz,Matthew Guevara,Tie Wang,Manasi Deshpande,Xiaofeng Wang,Zheng Li

DOI: https://doi.org/10.1145/3626772.3661370

2024-05-06

Abstract:In customer service technical support, swiftly and accurately retrieving relevant past issues is critical for efficiently resolving customer inquiries. The conventional retrieval methods in retrieval-augmented generation (RAG) for large language models (LLMs) treat a large corpus of past issue tracking tickets as plain text, ignoring the crucial intra-issue structure and inter-issue relations, which limits performance. We introduce a novel customer service question-answering method that amalgamates RAG with a knowledge graph (KG). Our method constructs a KG from historical issues for use in retrieval, retaining the intra-issue structure and inter-issue relations. During the question-answering phase, our method parses consumer queries and retrieves related sub-graphs from the KG to generate answers. This integration of a KG not only improves retrieval accuracy by preserving customer service structure information but also enhances answering quality by mitigating the effects of text segmentation. Empirical assessments on our benchmark datasets, utilizing key retrieval (MRR, Recall@K, NDCG@K) and text generation (BLEU, ROUGE, METEOR) metrics, reveal that our method outperforms the baseline by 77.6% in MRR and by 0.32 in BLEU. Our method has been deployed within LinkedIn's customer service team for approximately six months and has reduced the median per-issue resolution time by 28.6%.

Information Retrieval,Artificial Intelligence,Computation and Language,Machine Learning

What problem does this paper attempt to address?

The paper addresses the challenge of accurately and quickly retrieving relevant historical issues in customer technical support and proposes a retrieval-enhanced generation method combined with a knowledge graph (KG). Traditional methods treat a large number of historical issue tickets as plain text, ignoring the internal structure and relationships between issues, resulting in performance limitations. The new method improves retrieval accuracy by constructing a KG that preserves the question structure and relationships, and enhances answer quality by mitigating the impact of text segmentation. Experiments show that this method improves the baseline by 77.6% and 0.32 in terms of MRR and BLEU metrics respectively. After being applied in the customer service team at LinkedIn, it reduces the average problem resolution time by 28.6%.

Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering

Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering

Document Knowledge Graph to Enhance Question Answering with Retrieval Augmented Generation

KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation

WeKnow-RAG: An Adaptive Approach for Retrieval-Augmented Generation Integrating Web Search and Knowledge Graphs

Enhancing Question Answering for Enterprise Knowledge Bases using Large Language Models

Retrieval-Augmented Generation for Domain-Specific Question Answering: A Case Study on Pittsburgh and CMU

Enhancing Retrieval Augmented Generation Systems with Knowledge Graphs

Enhancing Retrieval and Managing Retrieval: A Four-Module Synergy for Improved Quality and Efficiency in RAG Systems

HybGRAG: Hybrid Retrieval-Augmented Generation on Textual and Relational Knowledge Bases

SimGRAG: Leveraging Similar Subgraphs for Knowledge Graphs Driven Retrieval-Augmented Generation

KG-Retriever: Efficient Knowledge Indexing for Retrieval-Augmented Large Language Models

RAG based Question-Answering for Contextual Response Prediction System

The Geometry of Queries: Query-Based Innovations in Retrieval-Augmented Generation

DR-RAG: Applying Dynamic Document Relevance to Retrieval-Augmented Generation for Question-Answering

Optimizing Retrieval-Augmented Generation with Elasticsearch for Enhanced Question-Answering Systems

G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering

Empowering Large Language Models to Set up a Knowledge Retrieval Indexer via Self-Learning

RuleRAG: Rule-guided retrieval-augmented generation with language models for question answering

Beyond-RAG: Question Identification and Answer Generation in Real-Time Conversations

HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction