Abstract:Retrieval-augmented generation (RAG) framework has shown promising potential in knowledge-intensive question answering (QA) by retrieving external corpus and generating based on augmented context. However, existing approaches only consider the query itself, neither specifying the retrieval preferences for the retrievers nor informing the generators of how to refer to the retrieved documents for the answers, which poses a significant challenge to the QA performance. To address these issues, we propose Rule-Guided Retrieval-Augmented Generation with LMs, which explicitly introduces symbolic rules as demonstrations for in-context learning (RuleRAG-ICL) to guide retrievers to retrieve logically related documents in the directions of rules and uniformly guide generators to generate answers attributed by the guidance of the same set of rules. Moreover, the combination of queries and rules can be further used as supervised fine-tuning data to update retrievers and generators (RuleRAG-FT) to achieve better rule-based instruction following capability, leading to retrieve more supportive results and generate more acceptable answers. To emphasize the attribution of rules, we construct five rule-aware QA benchmarks, including three temporal and two static scenarios, and equip RuleRAG with several kinds of retrievers and generators. Experiments demonstrate that training-free RuleRAG-ICL effectively improves the retrieval quality of +89.2% in Recall@10 scores and generation accuracy of +103.1% in exact match scores over standard RAG on average across the five benchmarks, and further fine-tuned RuleRAG-FT consistently yields more significant performance enhancement. Extensive analyses indicate that RuleRAG scales well with increasing numbers of retrieved documents and exhibits generalization ability for untrained rules.

RQE: Rule-Driven Query Expansion to Solve Empty Answers in SPARQL

Towards Empty Answers in SPARQL: Approximating Querying with RDF Embedding

Improving the Precision of RDF Question/Answering Systems: A Why Not Approach

Answering Why-Not Questions on SPARQL Queries

Natural Language Question Answering Over Rdf - A Graph Data Driven Approach

Expand, Rerank, and Retrieve: Query Reranking for Open-Domain Question Answering.

Query Answering with Inconsistent Existential Rules under Stable Model Semantics

Answering Natural Language Questions by Subgraph Matching over Knowledge Graphs

Increasing the LLM Accuracy for Question Answering: Ontologies to the Rescue!

Interactive Natural Language Question Answering over Knowledge Graphs

How To Build Templates For Rdf Question/Answering-An Uncertain Graph Similarity Join Approach

A State-transition Framework to Answer Complex Questions over Knowledge Base

Self-supervised BGP-graph Reasoning Enhanced Complex KBQA Via SPARQL Generation

Formal Query Generation for Question Answering over Knowledge Bases

RuleRAG: Rule-guided retrieval-augmented generation with language models for question answering

Improving Question Answering Based on Query Expansion with Wikipedia

A Universal Question-Answering Platform for Knowledge Graphs

NLP-Based Rule Learning from Legal Text for Question Answering

Learning Approximation Sets for Exploratory Queries

Natural Language Question Answering over RDF Data