ReliCD: A Reliable Cognitive Diagnosis Framework with Confidence Awareness

Yunfei Zhang,Chuan Qin,Dazhong Shen,Haiping Ma,Le Zhang,Xingyi Zhang,Hengshu Zhu

2023-12-29

Abstract:During the past few decades, cognitive diagnostics modeling has attracted increasing attention in computational education communities, which is capable of quantifying the learning status and knowledge mastery levels of students. Indeed, the recent advances in neural networks have greatly enhanced the performance of traditional cognitive diagnosis models through learning the deep representations of students and exercises. Nevertheless, existing approaches often suffer from the issue of overconfidence in predicting students' mastery levels, which is primarily caused by the unavoidable noise and sparsity in realistic student-exercise interaction data, severely hindering the educational application of diagnostic feedback. To address this, in this paper, we propose a novel Reliable Cognitive Diagnosis(ReliCD) framework, which can quantify the confidence of the diagnosis feedback and is flexible for different cognitive diagnostic functions. Specifically, we first propose a Bayesian method to explicitly estimate the state uncertainty of different knowledge concepts for students, which enables the confidence quantification of diagnostic feedback. In particular, to account for potential differences, we suggest modeling individual prior distributions for the latent variables of different ability concepts using a pre-trained model. Additionally, we introduce a logical hypothesis for ranking confidence levels. Along this line, we design a novel calibration loss to optimize the confidence parameters by modeling the process of student performance prediction. Finally, extensive experiments on four real-world datasets clearly demonstrate the effectiveness of our ReliCD framework.

Computers and Society,Machine Learning

What problem does this paper attempt to address?

The paper primarily addresses existing issues in the field of cognitive diagnosis by proposing a new framework—Reliable Cognitive Diagnosis (ReliCD) framework. Its core aim is to tackle the overconfidence problem present in current cognitive diagnosis models when predicting students' knowledge mastery levels, especially due to the inevitable noise and sparsity in real-world student-exercise interaction data.

### Research Problems

1. **Overconfidence Problem**: Existing cognitive diagnosis models tend to be overly confident when predicting students' mastery of specific knowledge points. That is, the diagnostic feedback provided may not be accurate, yet the model exhibits high certainty. This overconfidence severely limits the reliability of diagnostic feedback in practical online education systems.

2. **Noise and Sparsity**: Real-world student-exercise interaction data generally contain noise and sparsity, leading to inaccuracies and unreliability in estimating students' ability states with existing methods.

### Solutions

To address the above challenges, the paper proposes the ReliCD framework, which includes the following key points:

1. **Uncertainty Quantification**: Explicitly estimate the uncertainty of students' states on different knowledge points using Bayesian methods, allowing for the quantification of the confidence in diagnostic feedback. Gaussian latent variables are used to represent students' states, where the mean parameter indicates the average ability state, and the variance is used to quantify the confidence in the diagnostic feedback.

2. **Individual Prior Distribution**: Considering the potential differences between various ability concepts, the paper suggests using a pre-trained model to model the individual prior distribution for latent variables, better adapting to different ability concepts.

3. **Logical Assumption and Calibration Loss**: Introduce a logical assumption to rank confidence levels and design a new calibration loss to optimize the confidence parameters. By simulating the student performance prediction process, these parameters are optimized to ensure the reliability of the diagnostic feedback.

4. **Experimental Validation**: Extensive experiments on four real-world datasets demonstrate that the ReliCD framework is not only effective but also flexible, making it applicable to different cognitive diagnosis functions.

In summary, the paper aims to improve the reliability of cognitive diagnosis by proposing the ReliCD framework, ensuring that accurate and reliable diagnostic feedback can be provided even in the presence of noisy and sparse data.