Explainable AI needs formal notions of explanation correctness

Stefan Haufe,Rick Wilming,Benedict Clark,Rustam Zhumagambetov,Danny Panknin,Ahcène Boubekki

2024-09-26

Abstract:The use of machine learning (ML) in critical domains such as medicine poses risks and requires regulation. One requirement is that decisions of ML systems in high-risk applications should be human-understandable. The field of "explainable artificial intelligence" (XAI) seemingly addresses this need. However, in its current form, XAI is unfit to provide quality control for ML; it itself needs scrutiny. Popular XAI methods cannot reliably answer important questions about ML models, their training data, or a given test input. We recapitulate results demonstrating that popular XAI methods systematically attribute importance to input features that are independent of the prediction target. This limits their utility for purposes such as model and data (in)validation, model improvement, and scientific discovery. We argue that the fundamental reason for this limitation is that current XAI methods do not address well-defined problems and are not evaluated against objective criteria of explanation correctness. Researchers should formally define the problems they intend to solve first and then design methods accordingly. This will lead to notions of explanation correctness that can be theoretically verified and objective metrics of explanation performance that can be assessed using ground-truth data.

Machine Learning,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the deficiencies of current Explainable Artificial Intelligence (XAI) methods in terms of quality control and explanation correctness. Specifically:

1. **Limitations of XAI methods**: Existing XAI methods are unable to reliably answer important questions about machine - learning models, training data, or specific test inputs. For example, popular methods often assign importance scores to features that are irrelevant to the prediction target, which limits their application in model validation, data validation, model improvement, and scientific discovery.

2. **Lack of formal definitions**: Current XAI methods are not designed for well - defined problems, nor are they evaluated for the correctness of explanations according to objective criteria. This means that these methods lack theoretical verification and are difficult to evaluate their performance with real - data.

3. **Potential risks in key areas**: When using machine - learning systems in high - risk fields such as medicine, it is required that the decisions of these systems must be understandable to humans. However, current XAI methods are not sufficient to provide such assurance, so more stringent quality control is required.

### Main contributions of the paper

To address these problems, the author makes the following suggestions:

- **Formalize problem definitions**: Researchers should first clearly define the problems they want to solve and then design corresponding methods. This will help to establish the concept of explanation correctness that can be theoretically verified and provide objective indicators that can be evaluated with real - data.

- **Introduce the concept of causal variables**: The paper shows through two minimal examples how suppressor variables affect the performance of existing feature - attribution methods. Suppressor variables are those that do not have predictive power themselves but can improve the prediction effect. Research shows that many popular XAI methods will wrongly assign importance to these suppressor variables.

- **Propose improvement directions**: To solve the above problems, the author suggests developing new XAI methods, which should follow six steps:
1. Evaluate the information needs of users and stakeholders in specific application scenarios.
2. Define the formal requirements and XAI problems that meet these needs.
3. Design methods suitable for solving specific XAI problems.
4. Conduct theoretical analysis to ensure compliance with formal requirements.
5. Conduct empirical verification using appropriate benchmark datasets.
6. Improve the method according to further requirements.

Through these measures, the paper aims to promote the development of the XAI field so that it can better serve the quality assurance needs in practical applications.

### Summary

This paper emphasizes the limitations of current XAI methods in terms of explanation correctness and quality control and proposes directions for improvement. By formalizing problem definitions and introducing the concept of causal variables, researchers believe that more reliable and effective XAI tools can be developed, thereby increasing the credibility and transparency of machine - learning systems in high - risk applications.