Improving Reading Comprehension Question Generation with Data Augmentation and Overgenerate-and-rank

Nischal Ashok Kumar,Nigel Fernandez,Zichao Wang,Andrew Lan

DOI: https://doi.org/10.48550/arXiv.2306.08847

2023-06-15

Computation and Language

Abstract:Reading comprehension is a crucial skill in many aspects of education, including language learning, cognitive development, and fostering early literacy skills in children. Automated answer-aware reading comprehension question generation has significant potential to scale up learner support in educational activities. One key technical challenge in this setting is that there can be multiple questions, sometimes very different from each other, with the same answer; a trained question generation method may not necessarily know which question human educators would prefer. To address this challenge, we propose 1) a data augmentation method that enriches the training dataset with diverse questions given the same context and answer and 2) an overgenerate-and-rank method to select the best question from a pool of candidates. We evaluate our method on the FairytaleQA dataset, showing a 5% absolute improvement in ROUGE-L over the best existing method. We also demonstrate the effectiveness of our method in generating harder, "implicit" questions, where the answers are not contained in the context as text spans.

What problem does this paper attempt to address?

The paper aims to address several key challenges in the automatic question generation for reading comprehension. Specifically, it focuses on how to generate high-quality questions that align with educators' preferences given a context and an answer. The main challenges include:

1. **Diverse Question Selection**: For the same context and answer, there may be multiple valid questions. Existing question generation systems may not be able to identify which question is most preferred by educators.
2. **Generating More Challenging Questions**: Besides generating questions with answers explicitly contained in the context, generating implicit questions that require inferring the answer from the context is also a challenge.

To tackle these challenges, the paper proposes the following methods:

1. **Data Augmentation Method**: By synthesizing and generating diverse and relevant questions to enrich the training dataset, thereby improving the model's ability to recognize different types of questions.
2. **Over-Generation and Ranking Method**: Generating multiple candidate questions and then selecting the best question through a ranking model to ensure the generated questions better align with educators' preferences.

The paper validates the effectiveness of these methods through experiments on the FairytaleQA dataset. The results show that the proposed methods significantly improve the ROUGE-L metric compared to the existing best baseline methods, especially in generating implicit questions.