Abstract:Despite recent advancements, like Large Language Models (LLMs), in Question Answering with Machine Reading (QAMR), improving the factuality and faithfulness of QAMR models remains a significant challenge. QAMR models require both language knowledge and world knowledge to answer questions. Language knowledge encompasses syntax, semantics, pragmatics, and other language-specific elements. The extent of language knowledge reflects the model's language understanding capabilities. World knowledge, which refers to people's cognition of the world, may be parameterized knowledge of the pre-trained language models or textual knowledge of passages. We conduct a comparative study on these two kinds of knowledge and find that language knowledge is stable, while only part of world knowledge is stable and reliable. This motivates us to utilize textual knowledge of passages and avoid parameterized unstable world knowledge of pre-trained language models for QAMR task. To this end, this paper introduces the concept of Answerable without relying on unstable world knowledge external to the passage (AUKE) to determine whether a question can be answered without using parameterized unstable world knowledge of pre-trained language models. . We then define evidence as the simplest substring in the passage that supports AUKE. Based on evidence, , we introduce a novel faithfulness metric for the QAMR task. We propose a methodology that combines automated processes with manual refinement to augment QAMR datasets with evidence annotations to facilitate faithfulness evaluations. We apply this method to the Chinese QAMR dataset CMRC 2018 and DRCD to extend two datasets that support evidence-based faithfulness evaluation, CMRCFF (CMRC with Faithfulness) and DRCDFF (CMRC with Faithfulness). To alleviate the potential factuality and faithfulness issues induced by unstable world knowledge, we propose a method called Maximizing Discrimination Masking (MDM), which masks the word with the highest degree of distinguishability. MDM is an approximation method designed to circumvent the reliance on parameterized unstable world knowledge embedded within pre-trained language models utilized by QAMR systems. We conduct experiments under the fine-tune setting and few-shot setting on CMRCFF and DRCDFF. The results verify that our MDM approach can effectively improve the factuality and faithfulness of the models.

A Novel MRC Framework for Evidence Extracts in Judgment Documents

Evidence Sentence Extraction for Machine Reading Comprehension

An MRC Framework for Semantic Role Labeling

An End-to-end Joint Model for Evidence Information Extraction from Court Record Document.

A Feedback-Enhanced Two-Stage Framework for judicial machine reading comprehension

A Self-Training Method for Machine Reading Comprehension with Soft Evidence Extraction

EviDR: Evidence-Emphasized Discrete Reasoning for Reasoning Machine Reading Comprehension

An Interpretable Model Using Evidence Information for Multi-Hop Question Answering Over Long Texts

Multi-grained Evidence Inference for Multi-choice Reading Comprehension

Establish Evidence Chain Model on Chinese Criminal Judgment Documents Using Text Similarity Measure.

U3E: Unsupervised and Erasure-based Evidence Extraction for Machine Reading Comprehension

Maximizing Discrimination Masking for Faithful Question Answering with Machine Reading

Various Legal Factors Extraction Based on Machine Reading Comprehension.

Jointly Extracting Interventions, Outcomes, and Findings from RCT Reports with LLMs

Build Evidence Chain Relational Model Based on Chinese Judgment Documents.

ExpMRC: Explainability Evaluation for Machine Reading Comprehension

Enhancing cross-evidence reasoning graph for document-level relation extraction

KS-LLM: Knowledge Selection of Large Language Models with Evidence Document for Question Answering

Chain-of-Discussion: A Multi-Model Framework for Complex Evidence-Based Question Answering

R$^2$F: A General Retrieval, Reading and Fusion Framework for Document-level Natural Language Inference

Progressive Evidence Refinement for Open-domain Multimodal Retrieval Question Answering