Abstract:Machine Reading Comprehension (MRC) has achieved impressive answer inference performance in recent years but rarely considers the trustworthiness and reliability of the deployed systems. However, it is crucial to estimate the predictive uncertainty in real-world applications to measure how likely the prediction is wrong. Hence it is possible to abstain from the uncertain prediction with low confidence and build a trustworthy system. Prior studies use post-processing ways to measure the predictive uncertainty, such as employing heuristic softmax probability or training a calibrator on top of a trained MRC model. However, they only calibrate the confidence without considering the domain adaptation relationship. To handle the limitations, this paper presents TrustMRC, a non-postprocessing trustworthy MRC system that leverages (1) conditional calibration strategy to get reliable uncertainty, and (2) conditional adversarial learning strategy to learn transfer representations under domain shift setting. On the one hand, to estimate the predictive uncertainty, a conditional calibration module is proposed to predict whether the output of the answer prediction module is correct, and it is combined with an additional ECE constraint to restrict the confidence more reliable. On the other hand, for domain shift, TrustMRC designs a conditional adversarial learning strategy to learn transfer representations through a domain discriminator with uncertainty constraints, which takes both input and uncertainty alignment into account. Besides, TrustMRC is a non-postprocessing model that completes the answer prediction and uncertainty prediction in an end-to-end framework, so that these two sub-tasks can benefit from each other via multi-task learning. Instead of traditional EM and F1 metrics, EM-coverage and F1-coverage curves are used, for the trustworthiness-aware MRC evaluation. The experimental results on SQuAD 1.1, Natural Questions, and NewsQA datasets indicate that TrustMRC can make reliable predictions under domain shift settings.

Learning Invariant Representation Improves Robustness for MRC Models

The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Models

A Robust Adversarial Training Approach to Machine Reading Comprehension

Enhancing Robustness of Retrieval-Augmented Language Models with In-Context Learning

Robust Reading Comprehension with Linguistic Constraints Via Posterior Regularization.

Using calibrator to improve robustness in Machine Reading Comprehension

Defending Machine Reading Comprehension against Question-Targeted Attacks.

Robustness-Eva-MRC: Assessing and Analyzing the Robustness of Neural Models in Extractive Machine Reading Comprehension

Improving the robustness of machine reading comprehension model with hierarchical knowledge and auxiliary unanswerability prediction

Improving Lexical Embeddings for Robust Question Answering

ExpMRC: Explainability Evaluation for Machine Reading Comprehension

Trustworthy machine reading comprehension with conditional adversarial calibration

Leveraging Large Language Models for Multiple Choice Question Answering

Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models

A Multi-Task Learning Machine Reading Comprehension Model for Noisy Document (student Abstract)

Multilingual Multi-Aspect Explainability Analyses on Machine Reading Comprehension Models

Enhancing Pre-Trained Generative Language Models with Question Attended Span Extraction on Machine Reading Comprehension

Improving the Robustness of Deep Reading Comprehension Models by Leveraging Syntax Prior.

Multilingual Augmentation for Robust Visual Question Answering in Remote Sensing Images

Evaluating the Robustness of Machine Reading Comprehension Models to Low Resource Entity Renaming

Robust video question answering via contrastive cross-modality representation learning