PubMedQA: A Dataset for Biomedical Research Question Answering

Qiao Jin,Bhuwan Dhingra,Zhengping Liu,William W. Cohen,Xinghua Lu

2019-09-13

Abstract:We introduce PubMedQA, a novel biomedical question answering (QA) dataset collected from PubMed abstracts. The task of PubMedQA is to answer research questions with yes/no/maybe (e.g.: Do preoperative statins reduce atrial fibrillation after coronary artery bypass grafting?) using the corresponding abstracts. PubMedQA has 1k expert-annotated, 61.2k unlabeled and 211.3k artificially generated QA instances. Each PubMedQA instance is composed of (1) a question which is either an existing research article title or derived from one, (2) a context which is the corresponding abstract without its conclusion, (3) a long answer, which is the conclusion of the abstract and, presumably, answers the research question, and (4) a yes/no/maybe answer which summarizes the conclusion. PubMedQA is the first QA dataset where reasoning over biomedical research texts, especially their quantitative contents, is required to answer the questions. Our best performing model, multi-phase fine-tuning of BioBERT with long answer bag-of-word statistics as additional supervision, achieves 68.1% accuracy, compared to single human performance of 78.0% accuracy and majority-baseline of 55.2% accuracy, leaving much room for improvement. PubMedQA is publicly available at <a class="link-external link-https" href="https://pubmedqa.github.io" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Machine Learning,Quantitative Methods

What problem does this paper attempt to address?

The paper aims to address issues in the question-answering systems within the biomedical research field, with specific objectives including:

1. **Constructing a large-scale high-quality dataset**: Existing biomedical question-answering datasets are relatively small and mostly consist of simple factual questions, which are insufficient for evaluating models on complex reasoning tasks. Therefore, the paper proposes a new dataset named PubMedQA, which includes a large number of research questions and their corresponding answers. These questions require complex reasoning over literature abstracts to be answered.

2. **Promoting research on complex reasoning capabilities**: Unlike questions in other datasets that are mostly simple factual questions, the questions in PubMedQA require models to understand and reason out the answers, especially when dealing with quantitative information. This helps to advance the capabilities of natural language processing models in understanding and reasoning over scientific literature.

3. **Providing benchmark tests and pre-training resources**: The paper not only provides an annotated dataset (PQA-L) but also a large amount of unannotated data (PQA-U) and automatically generated question instances (PQA-A). These resources can be used for model training, semi-supervised learning, and pre-training, thereby providing foundational support for subsequent research.

In summary, the paper primarily addresses the issue of how to construct a large-scale question-answering dataset capable of evaluating models' complex reasoning abilities in biomedical literature and provides corresponding experimental setups and baseline models.