Diversify-verify-adapt: Efficient and Robust Retrieval-Augmented Ambiguous Question Answering

Yeonjun In,Sungchul Kim,Ryan A. Rossi,Md Mehrab Tanjim,Tong Yu,Ritwik Sinha,Chanyoung Park

2024-09-04

Abstract:The retrieval augmented generation (RAG) framework addresses an ambiguity in user queries in QA systems by retrieving passages that cover all plausible interpretations and generating comprehensive responses based on the passages. However, our preliminary studies reveal that a single retrieval process often suffers from low quality results, as the retrieved passages frequently fail to capture all plausible interpretations. Although the iterative RAG approach has been proposed to address this problem, it comes at the cost of significantly reduced efficiency. To address these issues, we propose the diversify-verify-adapt (DIVA) framework. DIVA first diversifies the retrieved passages to encompass diverse interpretations. Subsequently, DIVA verifies the quality of the passages and adapts the most suitable approach tailored to their quality. This approach improves the QA systems accuracy and robustness by handling low quality retrieval issue in ambiguous questions, while enhancing efficiency.

Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the low - quality retrieval and inefficiency in handling ambiguous questions in open - domain question - answering systems. Specifically:

1. **Low - quality retrieval**: The existing Retrieval - Augmented Generation (RAG) framework often fails to obtain high - quality paragraphs covering all possible interpretations in a single retrieval process, which leads to a significant decline in factual accuracy.
2. **Inefficiency**: Although the iterative RAG method can partially solve the low - quality retrieval problem, this method significantly increases the computational overhead, resulting in reduced efficiency.

To solve these problems, the paper proposes a new framework named **Diversify - Validate - Adapt (DIVA)**. DIVA improves the accuracy and efficiency of handling ambiguous questions through the following two key components:

- **Retrieval Diversification (RD)**: By inferring the pseudo - interpretations of the question and using these pseudo - interpretations to retrieve paragraphs covering multiple interpretations, the retrieval quality is improved.
- **Adaptive Generation (AG)**: Before using the retrieved paragraphs, the quality of these paragraphs is first verified, and the most appropriate generation strategy is selected according to the paragraph quality.

Through these methods, DIVA not only improves the accuracy and robustness of the question - answering system but also solves the deficiencies of the existing RAG framework in handling ambiguous questions while maintaining high efficiency.