Abstract:The advancement of machine learning (ML) models has led to the development of ML-based approaches to improve numerous software engineering tasks in software maintenance and evolution. Nevertheless, research indicates that despite their potential successes, ML models may not be employed in real-world scenarios because they often remain a black box to practitioners, lacking explainability in their reasoning. Recently, various rule-based model-agnostic Explainable AI (XAI) techniques, such as PyExplainer and LIME, have been employed to explain the predictions of ML models in software analytics tasks. This paper assesses the ability of these techniques (e.g., PyExplainer and LIME) to generate reliable and consistent explanations for ML models across various software analytics tasks, including Just-in-Time (JIT) defect prediction, clone detection, and the classification of useful code review comments. Our manual investigations find inconsistencies and anomalies in the explanations generated by these techniques. Therefore, we design a novel framework: Evaluation of Explainable AI (EvaluateXAI), along with granular-level evaluation metrics, to automatically assess the effectiveness of rule-based XAI techniques in generating reliable and consistent explanations for ML models in software analytics tasks. After conducting in-depth experiments involving seven state-of-the-art ML models trained on five datasets and six evaluation metrics, we find that none of the evaluation metrics reached 100\%, indicating the unreliability of the explanations generated by XAI techniques. Additionally, PyExplainer and LIME failed to provide consistent explanations for 86.11% and 77.78% of the experimental combinations, respectively. Therefore, our experimental findings emphasize the necessity for further research in XAI to produce reliable and consistent explanations for ML models in software analytics tasks.

Enhancing Transparency and Interpretability in Toxic Comment Classification: A Study on the Integration of Explainable Artificial Intelligence (XAI) Techniques || Dr. Pallavi Devendra Tawde, Mr. Jadyn Dias

ToxCCIn: Toxic Content Classification with Interpretability

Purging the Poison: A Machine Learning Approach to Filtering Toxic Comments

Explainable AI for Tuberculosis Detection using Deep Learning

ToxiSpanSE: An Explainable Toxicity Detection in Code Review Comments

Interpretability and Transparency of Machine Learning in File Fragment Analysis with Explainable Artificial Intelligence

RECAST: Enabling User Recourse and Interpretability of Toxicity Detection Models with Interactive Visualization

Interpretable Deep Learning Models: Enhancing Transparency and Trustworthiness in Explainable AI

Modeling subjectivity (by Mimicking Annotator Annotation) in toxic comment identification across diverse communities

Investigating Bias In Automatic Toxic Comment Detection: An Empirical Study

ToxVis: Enabling Interpretability of Implicit vs. Explicit Toxicity Detection Models with Interactive Visualization

Designing Toxic Content Classification for a Diversity of Perspectives

Toxic Comment Classification based on Personality Traits Using NLP

Explainable AI for Cloud-Based Machine Learning Interpretable Models and Transparency in Decision Making.

A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI

Diagnosis of Acute Poisoning Using Explainable Artificial Intelligence

More Questions than Answers? Lessons from Integrating Explainable AI into a Cyber-AI Tool

Unveiling the Black Box: A Comprehensive Review of Explainable AI Techniques

A Survey of Toxic Comment Classification Methods

Reading Between the Demographic Lines: Resolving Sources of Bias in Toxicity Classifiers

EvaluateXAI: A Framework to Evaluate the Reliability and Consistency of Rule-based XAI Techniques for Software Analytics Tasks