Abstract:Prior work has uncovered a set of common problems in state-of-the-art context-based question answering (QA) systems: a lack of attention to the context when the latter conflicts with a model's parametric knowledge, little robustness to noise, and a lack of consistency with their answers. However, most prior work focus on one or two of those problems in isolation, which makes it difficult to see trends across them. We aim to close this gap, by first outlining a set of -- previously discussed as well as novel -- desiderata for QA models. We then survey relevant analysis and methods papers to provide an overview of the state of the field. The second part of our work presents experiments where we evaluate 15 QA systems on 5 datasets according to all desiderata at once. We find many novel trends, including (1) systems that are less susceptible to noise are not necessarily more consistent with their answers when given irrelevant context; (2) most systems that are more susceptible to noise are more likely to correctly answer according to a context that conflicts with their parametric knowledge; and (3) the combination of conflicting knowledge and noise can reduce system performance by up to 96%. As such, our desiderata help increase our understanding of how these models work and reveal potential avenues for improvements.

Controllable Decontextualization of Yes/No Question and Answers into Factual Statements

Chapter 11. The meaning of question words in statements in child Mandarin

Overcoming Language Priors In Vqa Via Decomposed Linguistic Representations

An analysis of focus and its role in the answering systems of polar questions in Chinese and English

Question Answering System Of Confucian Analects Based On Pragmatics Information And Categories

Does the "most sinfully decadent cake ever" taste good? Answering Yes/No Questions from Figurative Contexts

Non-prototypical Interrogatives in Chinese

Interpreting Answers to Yes-No Questions in User-Generated Content

Triviality and interrogative embedding: context sensitivity, factivity, and neg-raising

Language style and recognition of the answers in health Q&A community: Moderating effects of medical terminology

Interpreting Answers to Yes-No Questions in Dialogues from Multiple Domains

Interactive Question Answering Based On Faq

Chinese natural language interrogative sentence semantization knowledge base automatic question-answering method

Effect of Using Pragmatics Information on Question Answering System of Analects of Confucius

Why Did the Chicken Cross the Road? Rephrasing and Analyzing Ambiguous Questions in VQA

Desiderata for the Context Use of Question Answering Systems

ControversialQA: Exploring Controversy in Question Answering

An action-specific examination of the role of answerers' gaze orientation in managing transition relevance

Knowledge-Oriented Answer Extraction in Chinese Question Answering System for E-Learning

Real Environment Oriented Question Analyzing

Collocation Polarity Disambiguation Using Web-based Pseudo Contexts.