DT-QDC: A Dataset for Question Comprehension in Online Test.

Sijin Wu,Yujiu Yang,Nicholas Yung,Zhengchen Shen,Zeyang Lei

DOI: https://doi.org/10.18653/v1/2020.coling-main.569

2020-01-01

Abstract:As education switches from the traditional classroom environment to online education and assessment, teachers could not follow students' learning behavior closely which leads to inaccurate assessment of students' knowledge.A well-defined method to precisely measure difficulty of questions is critical as it can guide teachers in assessing students' performance and help provide customized attention to students.In this paper, we explore the concept of question difficulty and share our new Chinese Driving Test Question Difficulty Comprehension (DT-QDC) dataset.Different from the existing question datasets, we mainly design for question evaluation and comprehension in online testing, so each question has enriched attributes and difficulty label.Additional attributes such as keywords, chapter, and question type would allow models to understand questions more precisely.We proposed the Ordinal Regression Multi-Source BERT (ORMS-BERT) model to assess the difficulty of the question, which outperforms different baselines by 6.77% on F1-score and 15.92% on MAE, 28.26% on MSE on the DT-QDC dataset, laying the foundation for the question difficulty comprehension task.