Predictability and Comprehensibility in Post-Hoc XAI Methods: A User-Centered Analysis

Anahid Jalali,Bernhard Haslhofer,Simone Kriglstein,Andreas Rauber

2023-09-21

Abstract:Post-hoc explainability methods aim to clarify predictions of black-box machine learning models. However, it is still largely unclear how well users comprehend the provided explanations and whether these increase the users ability to predict the model behavior. We approach this question by conducting a user study to evaluate comprehensibility and predictability in two widely used tools: LIME and SHAP. Moreover, we investigate the effect of counterfactual explanations and misclassifications on users ability to understand and predict the model behavior. We find that the comprehensibility of SHAP is significantly reduced when explanations are provided for samples near a model's decision boundary. Furthermore, we find that counterfactual explanations and misclassifications can significantly increase the users understanding of how a machine learning model is making decisions. Based on our findings, we also derive design recommendations for future post-hoc explainability methods with increased comprehensibility and predictability.

Machine Learning,Artificial Intelligence,Human-Computer Interaction

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to explore and evaluate the effectiveness and limitations of post - hoc Explainable Artificial Intelligence (XAI) methods in helping users understand and predict the behavior of black - box machine - learning models. Specifically, the author answers the following key questions through user studies:

1. **How well do users understand the explanations provided by different XAI methods?**
- Researchers selected two widely - used post - hoc interpretability tools - LIME (Local Interpretable Model - agnostic Explanations) and SHAP (Shapley Additive Explanations), and evaluated users' ability to understand these explanations through user studies.

2. **Can users predict the model's decisions based on the provided explanations?**
- The author further studied whether users can use these explanations to predict the behavior of the model, especially when facing new samples.

3. **Can counterfactual explanations and the visualization of misclassified samples improve users' prediction ability?**
- The study also explored how counterfactual explanations and the visualization of misclassified samples can help users understand the model's decisions, and whether these visualizations can improve users' prediction ability.

4. **Can the visualization of local XAI explanations guide users to find global explanations?**
- By collecting users' subjective feedback on the explanations, researchers hope to understand whether these explanations can help users derive global understanding from local explanations.

### Main findings

- **The comprehensibility of SHAP explanations significantly decreases near the decision boundary**: When the explanations are close to the model's decision boundary, SHAP explanations become difficult to understand.
- **Counterfactual explanations and misclassified samples can significantly improve users' understanding**: These additional forms of explanation help users better understand how the model makes decisions.
- **Users' prediction ability with LIME is higher than that with SHAP**: In some cases, users can predict better using LIME explanations.
- **Users' subjective feedback provides valuable insights for improving XAI methods**: For example, users' complaints about inconsistent explanations and different visualization scales indicate that these problems affect their understanding of the model's behavior.

### Design suggestions

Based on the above findings, the author proposes some design suggestions to improve future post - hoc interpretability methods to make them more understandable and predictive. These suggestions include but are not limited to:
- Provide more intuitive visualization tools to help users better understand feature contributions.
- Include more context information in the explanations, especially when dealing with complex or ambiguous situations.
- Develop interactive explanation tools that can adapt to the needs of different types of users.

Through these studies, the author hopes to promote the development of the XAI field, make interpretability tools more in line with users' needs, and ultimately improve the transparency and interpretability of machine - learning models.