Abstract:The advancement of machine learning (ML) models has led to the development of ML-based approaches to improve numerous software engineering tasks in software maintenance and evolution. Nevertheless, research indicates that despite their potential successes, ML models may not be employed in real-world scenarios because they often remain a black box to practitioners, lacking explainability in their reasoning. Recently, various rule-based model-agnostic Explainable AI (XAI) techniques, such as PyExplainer and LIME, have been employed to explain the predictions of ML models in software analytics tasks. This paper assesses the ability of these techniques (e.g., PyExplainer and LIME) to generate reliable and consistent explanations for ML models across various software analytics tasks, including Just-in-Time (JIT) defect prediction, clone detection, and the classification of useful code review comments. Our manual investigations find inconsistencies and anomalies in the explanations generated by these techniques. Therefore, we design a novel framework: Evaluation of Explainable AI (EvaluateXAI), along with granular-level evaluation metrics, to automatically assess the effectiveness of rule-based XAI techniques in generating reliable and consistent explanations for ML models in software analytics tasks. After conducting in-depth experiments involving seven state-of-the-art ML models trained on five datasets and six evaluation metrics, we find that none of the evaluation metrics reached 100\%, indicating the unreliability of the explanations generated by XAI techniques. Additionally, PyExplainer and LIME failed to provide consistent explanations for 86.11% and 77.78% of the experimental combinations, respectively. Therefore, our experimental findings emphasize the necessity for further research in XAI to produce reliable and consistent explanations for ML models in software analytics tasks.

VIME: Visual Interactive Model Explorer for Identifying Capabilities and Limitations of Machine Learning Models for Sequential Decision-Making

ViCE: Visual Counterfactual Explanations for Machine Learning Models

EvaluateXAI: A Framework to Evaluate the Reliability and Consistency of Rule-based XAI Techniques for Software Analytics Tasks

Explain To Decide: A Human-Centric Review on the Role of Explainable Artificial Intelligence in AI-assisted Decision Making

A Study on Multimodal and Interactive Explanations for Visual Question Answering

Explaining Any ML Model? -- On Goals and Capabilities of XAI

Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence

EXplainable Artificial Intelligence (XAI)—From Theory to Methods and Applications

LangXAI: Integrating Large Vision Models for Generating Textual Explanations to Enhance Explainability in Visual Perception Tasks

VitrAI -- Applying Explainable AI in the Real World

DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations

Stable local interpretable model-agnostic explanations based on a variational autoencoder

Explingo: Explaining AI Predictions using Large Language Models

X-VARS: Introducing Explainability in Football Refereeing with Multi-Modal Large Language Model

Interpretable Deep Learning Models: Enhancing Transparency and Trustworthiness in Explainable AI

ViTmiX: Vision Transformer Explainability Augmented by Mixed Visualization Methods

XAI for All: Can Large Language Models Simplify Explainable AI?

Do Explanations Reflect Decisions? A Machine-centric Strategy to Quantify the Performance of Explainability Algorithms

Unveiling the Black Box: A Comprehensive Review of Explainable AI Techniques

Explainable artificial intelligence: a comprehensive review

Fool Me Once? Contrasting Textual and Visual Explanations in a Clinical Decision-Support Setting