Abstract:In recent years, Explainable AI (xAI) attracted a lot of attention as various countries turned explanations into a legal right. xAI allows for improving models beyond the accuracy metric by, e.g., debugging the learned pattern and demystifying the AI's behavior. The widespread use of xAI brought new challenges. On the one hand, the number of published xAI algorithms underwent a boom, and it became difficult for practitioners to select the right tool. On the other hand, some experiments did highlight how easy data scientists could misuse xAI algorithms and misinterpret their results. To tackle the issue of comparing and correctly using feature importance xAI algorithms, we propose Compare-xAI, a benchmark that unifies all exclusive functional testing methods applied to xAI algorithms. We propose a selection protocol to shortlist non-redundant functional tests from the literature, i.e., each targeting a specific end-user requirement in explaining a model. The benchmark encapsulates the complexity of evaluating xAI methods into a hierarchical scoring of three levels, namely, targeting three end-user groups: researchers, practitioners, and laymen in xAI. The most detailed level provides one score per test. The second level regroups tests into five categories (fidelity, fragility, stability, simplicity, and stress tests). The last level is the aggregated comprehensibility score, which encapsulates the ease of correctly interpreting the algorithm's output in one easy to compare value. Compare-xAI's interactive user interface helps mitigate errors in interpreting xAI results by quickly listing the recommended xAI solutions for each ML task and their current limitations. The benchmark is made available at <a class="link-external link-https" href="https://karim-53.github.io/cxai/" rel="external noopener nofollow">this https URL</a>

Does Dataset Complexity Matters for Model Explainers?

How Reliable and Stable are Explanations of XAI Methods?

Beyond the Black Box: Do More Complex Deep Learning Models Provide Superior XAI Explanations?

Model Interpretation and Explainability: Towards Creating Transparency in Prediction Models

Explainability is NOT a Game

Explanation matters: An experimental study on explainable AI

Explaining Any ML Model? -- On Goals and Capabilities of XAI

BEExAI: Benchmark to Evaluate Explainable AI

Quantifying Explainability in Outcome-Oriented Predictive Process Monitoring

Which Neural Network Makes More Explainable Decisions? an Approach Towards Measuring Explainability

How Well Do Feature-Additive Explainers Explain Feature-Additive Predictors?

Distance-Restricted Explanations: Theoretical Underpinnings & Efficient Implementation

EXACT: Towards a platform for empirically benchmarking Machine Learning model explanation methods

Evaluating Explainability in Machine Learning Predictions through Explainer-Agnostic Metrics

Helpful, Misleading or Confusing: How Humans Perceive Fundamental Building Blocks of Artificial Intelligence Explanations

Evaluating the Explainability of Neural Rankers

A Comparative Analysis of Model Agnostic Techniques for Explainable Artificial Intelligence

Explainable AI without Interpretable Model

Do We Need Another Explainable AI Method? Toward Unifying Post-hoc XAI Evaluation Methods into an Interactive and Multi-dimensional Benchmark

Black Box Model Explanations and the Human Interpretability Expectations -- An Analysis in the Context of Homicide Prediction

Features of Explainability: How users understand counterfactual and causal explanations for categorical and continuous features in XAI