Knowledge Graph Based Agent for Complex, Knowledge-Intensive QA in Medicine

Xiaorui Su,Yibo Wang,Shanghua Gao,Xiaolong Liu,Valentina Giunchiglia,Djork-Arné Clevert,Marinka Zitnik

2024-10-07

Abstract:Biomedical knowledge is uniquely complex and structured, requiring distinct reasoning strategies compared to other scientific disciplines like physics or chemistry. Biomedical scientists do not rely on a single approach to reasoning; instead, they use various strategies, including rule-based, prototype-based, and case-based reasoning. This diversity calls for flexible approaches that accommodate multiple reasoning strategies while leveraging in-domain knowledge. We introduce KGARevion, a knowledge graph (KG) based agent designed to address the complexity of knowledge-intensive medical queries. Upon receiving a query, KGARevion generates relevant triplets by using the knowledge base of the LLM. These triplets are then verified against a grounded KG to filter out erroneous information and ensure that only accurate, relevant data contribute to the final answer. Unlike RAG-based models, this multi-step process ensures robustness in reasoning while adapting to different models of medical reasoning. Evaluations on four gold-standard medical QA datasets show that KGARevion improves accuracy by over 5.2%, outperforming 15 models in handling complex medical questions. To test its capabilities, we curated three new medical QA datasets with varying levels of semantic complexity, where KGARevion achieved a 10.4% improvement in accuracy.

Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the complex knowledge-intensive question answering (QA) problem in the medical field. Specifically, existing large language models (LLMs) have the following shortcomings when dealing with medical issues:

1. **Inaccurate information retrieval**: LLMs often make errors in retrieving medical information, miss key information, and are inconsistent with current scientific and medical knowledge.
2. **Lack of multi-source and fact-based knowledge integration capability**: LLMs find it difficult to systematically integrate different types of evidence, including structured, codified knowledge from formal research and non-codified knowledge such as expert experience.
3. **Difficulty adapting to domain-specific knowledge**: LLMs trained on general knowledge perform poorly when dealing with questions that require specialized medical knowledge.

To address these issues, the paper proposes KGAR EVION, a proxy model based on knowledge graphs (KG), which can generate relevant triples through a multi-step process and use existing KGs for validation to ensure information accuracy. Additionally, KGAR EVION outperforms 15 existing models on four standard medical QA datasets, with an accuracy improvement of over 5.2%, and achieves significant improvements on three newly created datasets, with an accuracy increase of 10.4%. The model can flexibly choose the most appropriate reasoning method for medical questions of varying difficulty levels, thereby improving the accuracy of complex medical QA tasks.