Abstract:Recent state-of-the-art open-domain QA models are typically based on a two stage retriever-reader approach in which the retriever first finds the relevant knowledge/passages and the reader then leverages that to predict the answer. Prior work has shown that the performance of the reader usually tends to improve with the increase in the number of these passages. Thus, state-of-the-art models use a large number of passages (e.g. 100) for inference. While the reader in this approach achieves high prediction performance, its inference is computationally very expensive. We humans, on the other hand, use a more efficient strategy while answering: firstly, if we can confidently answer the question using our already acquired knowledge then we do not even use the external knowledge, and in the case when we do require external knowledge, we don't read the entire knowledge at once, instead, we only read that much knowledge that is sufficient to find the answer. Motivated by this procedure, we ask a research question "Can the open-domain QA reader utilize external knowledge efficiently like humans without sacrificing the prediction performance?" Driven by this question, we explore an approach that utilizes both 'closed-book' (leveraging knowledge already present in the model parameters) and 'open-book' inference (leveraging external knowledge). Furthermore, instead of using a large fixed number of passages for open-book inference, we dynamically read the external knowledge in multiple 'knowledge iterations'. Through comprehensive experiments on NQ and TriviaQA datasets, we demonstrate that this dynamic reading approach improves both the 'inference efficiency' and the 'prediction accuracy' of the reader. Comparing with the FiD reader, this approach matches its accuracy by utilizing just 18.32% of its reader inference cost and also outperforms it by achieving up to 55.10% accuracy on NQ Open.

Frustratingly Hard Evidence Retrieval for QA Over Books

Narrative Question Answering with Cutting-Edge Open-Domain QA Techniques: A Comprehensive Study

Evidentiality-aware Retrieval for Overcoming Abstractiveness in Open-Domain Question Answering

NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens

Retrieving and Reading: A Comprehensive Survey on Open-domain Question Answering

Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehension

Evidence Aggregation for Answer Re-Ranking in Open-Domain Question Answering

Careful Selection of Knowledge to solve Open Book Question Answering

Examination-Style Reading Comprehension with Neural augmented Retrieval

Novel Knowledge-Based System with Relation Detection and Textual Evidence for Question Answering Research.

Improving Health Question Answering with Reliable and Time-Aware Evidence Retrieval

Assertion-based QA with Question-Aware Open Information Extraction

LogQA: Question Answering in Unstructured Logs

It is AI's Turn to Ask Humans a Question: Question-Answer Pair Generation for Children's Story Books

ReasonChainQA: Text-based Complex Question Answering with Explainable Evidence Chains

LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning

DetectiveQA: Evaluating Long-Context Reasoning on Detective Novels

Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?

When Retriever-Reader Meets Scenario-Based Multiple-Choice Questions

Evidence Sentence Extraction for Machine Reading Comprehension

Evaluating Open-QA Evaluation