Exploring the Relationship Between Feature Attribution Methods and Model Performance

Priscylla Silva,Claudio T. Silva,Luis Gustavo Nonato

2024-05-23

Abstract:Machine learning and deep learning models are pivotal in educational contexts, particularly in predicting student success. Despite their widespread application, a significant gap persists in comprehending the factors influencing these models' predictions, especially in explainability within education. This work addresses this gap by employing nine distinct explanation methods and conducting a comprehensive analysis to explore the correlation between the agreement among these methods in generating explanations and the predictive model's performance. Applying Spearman's correlation, our findings reveal a very strong correlation between the model's performance and the agreement level observed among the explanation methods.

Machine Learning

What problem does this paper attempt to address?

The paper discusses the relationship between feature attribution methods and the performance of machine learning and deep learning models, particularly in the field of education. The research reveals that these models are widely used for predicting student success, but there is a significant gap in understanding the predictive factors, particularly in terms of interpretability. The paper conducts a comprehensive analysis of the models using nine different explanation methods to explore the relationship between the consensus level in generating explanations and the predictive performance of the models.

The author calculates the (in)consistency measures between different explanation methods, such as feature consistency, sign consistency, rank consistency, and sign rank consistency, and uses Spearman correlation to study the relationship between these measures and the model performance (measured by AUC). Experimental results on two real-world datasets, one about an introductory programming course and another from a study by Amrieh et al., show that there is a strong positive correlation between model performance and the consistency of the explanation methods in most cases. When the model performance improves, the consensus among the explanation methods also increases, indicating that higher quality models are easier to interpret.

This finding is of great significance to practitioners in the field of education because choosing well-performing models can enhance the consistency and reliability among explanation methods, leading to a better understanding of the factors that influence student success. The research highlights the close connection between model performance and interpretability, suggesting that the model's performance should be taken into account before using explanation methods.