A Trustworthy View on Explainable Artificial Intelligence Method Evaluation.

Ding Li,Yan Liu,Jun Huang,Zerui Wang

DOI: https://doi.org/10.1109/mc.2022.3233806

2023-01-01

Computer

Abstract:In this article, we follow a process of explainable artificial intelligence (XAI) method development and define two metrics in terms of consistency and efficiency in guiding the evaluation of XAI explanations.

What problem does this paper attempt to address?

Exploring Evaluation Methodologies for Explainable AI: Guidelines for Objective and Subjective Assessment

Sule Tekkesinoglu

DOI: https://doi.org/10.2139/ssrn.4667052

2024-01-01

SSRN Electronic Journal

Abstract:This article explores the landscape of evaluation methodologies for Explainable Artificial Intelligence (XAI), focusing on both objective and subjective assessment paradigms. Objective evaluation methods provide structured and measurable assessments, offering insights into the system's performance, technical robustness, and functional effectiveness through quantitative metrics. These metrics are practical in benchmarking, verifying, and estimating functional effectiveness against predefined standards and existing explanation methods. Subjective assessments explore user-centric dimensions, elucidating perceptions, preferences, and interactions with AI-generated explanations. These assessment techniques unravel the human factors, capturing the user's experience of quality, satisfaction, trust, and performance. This work presents a list of evaluation metrics concerning objective and subjective properties of explanations. Additionally, concerning subjective measurements, an example of a user satisfaction questionnaire along with open-ended questions is provided. These can be tailored based on the context of the explanations and the user group being surveyed. Taken together, the outlined assessment methods provide a framework that caters to the diverse facets of XAI explainability to facilitate a deeper understanding of XAI systems' capabilities and enhance trust in AI-driven technologies.
Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial Intelligence

Sajid Ali,Tamer Abuhmed,Shaker El-Sappagh,Khan Muhammad,Jose M. Alonso-Moral,Roberto Confalonieri,Riccardo Guidotti,Javier Del Ser,Natalia Díaz-Rodríguez,Francisco Herrera

DOI: https://doi.org/10.1016/j.inffus.2023.101805

IF: 18.6

2023-04-21

Information Fusion

Abstract:Artificial intelligence (AI) is currently being utilized in a wide range of sophisticated applications, but the outcomes of many AI models are challenging to comprehend and trust due to their black-box nature. Usually, it is essential to understand the reasoning behind an AI model's decision-making. Thus, the need for eXplainable AI (XAI) methods for improving trust in AI models has arisen. XAI has become a popular research subject within the AI field in recent years. Existing survey papers have tackled the concepts of XAI, its general terms, and post-hoc explainability methods but there have not been any reviews that have looked at the assessment methods, available tools, XAI datasets, and so on. Therefore, in this comprehensive study, we provide readers with an overview of the current research and trends in this rapidly emerging area with a case study example. The review starts by explaining the background of XAI, common definitions, and summarizing recently proposed techniques in XAI for supervised machine learning. The review divides XAI techniques into four axes using a hierarchical categorization system: (i) data explainability, (ii) model explainability, (iii) post-hoc explainability, and (iv) assessment of explanations. We also introduce available evaluation metrics as well as open-source packages and datasets with future research directions. Then, the significance of explainability in terms of legal demands, user viewpoints, and application orientation is outlined, termed as XAI concerns. This paper advocates for tailoring explanation content to specific user types. An examination of XAI techniques and evaluation was conducted by looking at 410 critical articles, published between January 2016 and October 2022, in reputed journals and using a wide range of research databases as a source of information. The article is aimed at XAI researchers who are interested in making their AI models more trustworthy, as well as towards researchers from other disciplines who are looking for effective XAI methods to complete tasks with confidence while communicating meaning from data.

computer science, artificial intelligence, theory & methods
A New Perspective on Evaluation Methods for Explainable Artificial Intelligence (XAI)

Timo Speith,Markus Langer

2023-07-26

Abstract:Within the field of Requirements Engineering (RE), the increasing significance of Explainable Artificial Intelligence (XAI) in aligning AI-supported systems with user needs, societal expectations, and regulatory standards has garnered recognition. In general, explainability has emerged as an important non-functional requirement that impacts system quality. However, the supposed trade-off between explainability and performance challenges the presumed positive influence of explainability. If meeting the requirement of explainability entails a reduction in system performance, then careful consideration must be given to which of these quality aspects takes precedence and how to compromise between them. In this paper, we critically examine the alleged trade-off. We argue that it is best approached in a nuanced way that incorporates resource availability, domain characteristics, and considerations of risk. By providing a foundation for future research and best practices, this work aims to advance the field of RE for AI.

Artificial Intelligence,Software Engineering
The Analysis and Development of an XAI Process on Feature Contribution Explanation.

Jun Huang,Zerui Wang,Ding Li,Yan Liu

DOI: https://doi.org/10.1109/bigdata55660.2022.10020313

IF: 4.426

2022-01-01

Big Data

Abstract:Explainable Artificial Intelligence (XAI) research focuses on effective explanation techniques to understand and build AI models with trust, reliability, safety, and fairness. Feature importance explanation summarizes feature contributions for end-users to make model decisions. However, XAI methods may produce varied summaries that lead to further analysis to evaluate the consistency across multiple XAI methods on the same model and data set. This paper defines metrics to measure the consistency of feature contribution explanation summaries under feature importance order and saliency map. Driven by these consistency metrics, we develop an XAI process oriented on the XAI criterion of feature importance, which performs a systematical selection of XAI techniques and evaluation of explanation consistency. We demonstrate the process development involving twelve XAI methods on three topics, including a search ranking system, code vulnerability detection and image classification. Our contribution is a practical and systematic process with defined consistency metrics to produce rigorous feature contribution explanations.
EXplainable Artificial Intelligence (XAI)—From Theory to Methods and Applications

Evandro S. Ortigossa,Thales Gonçalves,Luis Gustavo Nonato

DOI: https://doi.org/10.1109/access.2024.3409843

IF: 3.9

2024-06-15

IEEE Access

Abstract:Intelligent applications supported by Machine Learning have achieved remarkable performance rates for a wide range of tasks in many domains. However, understanding why a trained algorithm makes a particular decision remains problematic. Given the growing interest in the application of learning-based models, some concerns arise in the dealing with sensible environments, which may impact users' lives. The complex nature of those models' decision mechanisms makes them the so-called "black boxes," in which the understanding of the logic behind automated decision-making processes by humans is not trivial. Furthermore, the reasoning that leads a model to provide a specific prediction can be more important than performance metrics, which introduces a trade-off between interpretability and model accuracy. Explaining intelligent computer decisions can be regarded as a way to justify their reliability and establish trust. In this sense, explanations are critical tools that verify predictions to discover errors and biases previously hidden within the models' complex structures, opening up vast possibilities for more responsible applications. In this review, we provide theoretical foundations of Explainable Artificial Intelligence (XAI), clarifying diffuse definitions and identifying research objectives, challenges, and future research lines related to turning opaque machine learning outputs into more transparent decisions. We also present a careful overview of the state-of-the-art explainability approaches, with a particular analysis of methods based on feature importance, such as the well-known LIME and SHAP. As a result, we highlight practical applications of the successful use of XAI.

computer science, information systems,telecommunications,engineering, electrical & electronic
Study on the Helpfulness of Explainable Artificial Intelligence

Tobias Labarta,Elizaveta Kulicheva,Ronja Froelian,Christian Geißler,Xenia Melman,Julian von Klitzing

DOI: https://doi.org/10.1007/978-3-031-63803-9_16

2024-10-14

Abstract:Explainable Artificial Intelligence (XAI) is essential for building advanced machine learning-powered applications, especially in critical domains such as medical diagnostics or autonomous driving. Legal, business, and ethical requirements motivate using effective XAI, but the increasing number of different methods makes it challenging to pick the right ones. Further, as explanations are highly context-dependent, measuring the effectiveness of XAI methods without users can only reveal a limited amount of information, excluding human factors such as the ability to understand it. We propose to evaluate XAI methods via the user's ability to successfully perform a proxy task, designed such that a good performance is an indicator for the explanation to provide helpful information. In other words, we address the helpfulness of XAI for human decision-making. Further, a user study on state-of-the-art methods was conducted, showing differences in their ability to generate trust and skepticism and the ability to judge the rightfulness of an AI decision correctly. Based on the results, we highly recommend using and extending this approach for more objective-based human-centered user studies to measure XAI performance in an end-to-end fashion.

Human-Computer Interaction,Artificial Intelligence
Explainable artificial intelligence: a comprehensive review

Dang Minh,H. Xiang Wang,Y. Fen Li,Tan N. Nguyen

DOI: https://doi.org/10.1007/s10462-021-10088-y

IF: 9.588

2021-11-18

Artificial Intelligence Review

Abstract:Thanks to the exponential growth in computing power and vast amounts of data, artificial intelligence (AI) has witnessed remarkable developments in recent years, enabling it to be ubiquitously adopted in our daily lives. Even though AI-powered systems have brought competitive advantages, the black-box nature makes them lack transparency and prevents them from explaining their decisions. This issue has motivated the introduction of explainable artificial intelligence (XAI), which promotes AI algorithms that can show their internal process and explain how they made decisions. The number of XAI research has increased significantly in recent years, but there lacks a unified and comprehensive review of the latest XAI progress. This review aims to bridge the gap by discovering the critical perspectives of the rapidly growing body of research associated with XAI. After offering the readers a solid XAI background, we analyze and review various XAI methods, which are grouped into (i) pre-modeling explainability, (ii) interpretable model, and (iii) post-modeling explainability. We also pay attention to the current methods that dedicate to interpret and analyze deep learning methods. In addition, we systematically discuss various XAI challenges, such as the trade-off between the performance and the explainability, evaluation methods, security, and policy. Finally, we show the standard approaches that are leveraged to deal with the mentioned challenges.

computer science, artificial intelligence
Metrics for Explainable AI: Challenges and Prospects

Robert R. Hoffman,Shane T. Mueller,Gary Klein,Jordan Litman

DOI: https://doi.org/10.48550/arXiv.1812.04608

2019-02-01

Abstract:The question addressed in this paper is: If we present to a user an AI system that explains how it works, how do we know whether the explanation works and the user has achieved a pragmatic understanding of the AI? In other words, how do we know that an explanainable AI system (XAI) is any good? Our focus is on the key concepts of measurement. We discuss specific methods for evaluating: (1) the goodness of explanations, (2) whether users are satisfied by explanations, (3) how well users understand the AI systems, (4) how curiosity motivates the search for explanations, (5) whether the user's trust and reliance on the AI are appropriate, and finally, (6) how the human-XAI work system performs. The recommendations we present derive from our integration of extensive research literatures and our own psychometric evaluations.

Artificial Intelligence
Explainability of Artificial Intelligence Methods, Applications and Challenges: A Comprehensive Survey

Weiping Ding,Mohamed Abdel-Basset,Hossam Hawash,Ahmed M. Ali

DOI: https://doi.org/10.1016/j.ins.2022.10.013

IF: 8.1

2022-01-01

Information Sciences

Abstract:The continuous advancement of Artificial Intelligence (AI) has been revolutionizing the strategy of decision-making in different life domains. Regardless of this achievement, AI algorithms have been built as Black-Boxes , that is as they hide their internal rationality and learning methodology from the human leaving many unanswered questions about how and why the AI decisions are made. The absence of explanation results in a sensible and ethical challenge. Explainable Artificial Intelligence (XAI) is an evolving subfield of AI that emphasizes developing a plethora of tools and techniques for unboxing the Black-Box AI solutions by generating human-comprehensible, insightful, and transparent explanations of AI decisions. This study begins by discussing the primary principles of XAI research, Black-Box problems, the targeted audience, and the related notion of explainability over the historical timeline of the XAI studies and accordingly establishes an innovative definition of explainability that addresses the earlier theoretical proposals. According to an extensive analysis of the literature, this study contributes to the body of knowledge by driving a fine-grained, multi-level, and multi-dimension taxonomy for insightful categorization of XAI studies with the main aim to shed light on the variations and commonalities of existing algorithms paving the way for extra methodological developments. Then, an experimental comparative analysis is presented for the explanation generated by common XAI algorithms applied to different categories of data to highlight their properties, advantages, and flaws. Followingly, this study discusses and categorizes the evaluation metrics for the XAI-generated explanation and the findings show that there is no common consensus on how an explanation must be expressed, and how its quality and dependability should be evaluated. The findings show that XAI can contribute to realizing responsible and trustworthy AI, however, the advantages of interpretability should be technically demonstrated, and complementary procedures and regulations are required to give actionable information that can empower decision-making in real-world applications. Finally, the tutorial is crowned by discussing the open research questions, challenges, and future directions that serve as a roadmap for the AI community to advance the research in XAI and to inspire specialists and practitioners to take the advantage of XAI in different disciplines.
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems

Sina Mohseni,Niloofar Zarei,Eric D. Ragan

DOI: https://doi.org/10.1145/3387166

IF: 3.4

2021-12-31

ACM Transactions on Interactive Intelligent Systems

Abstract:The need for interpretable and accountable intelligent systems grows along with the prevalence of artificial intelligence ( AI ) applications used in everyday life. Explainable AI ( XAI ) systems are intended to self-explain the reasoning behind system decisions and predictions. Researchers from different disciplines work together to define, design, and evaluate explainable systems. However, scholars from different disciplines focus on different objectives and fairly independent topics of XAI research, which poses challenges for identifying appropriate design and evaluation methodology and consolidating knowledge across efforts. To this end, this article presents a survey and framework intended to share knowledge and experiences of XAI design and evaluation methods across multiple disciplines. Aiming to support diverse design goals and evaluation methods in XAI research, after a thorough review of XAI related papers in the fields of machine learning, visualization, and human-computer interaction, we present a categorization of XAI design goals and evaluation methods. Our categorization presents the mapping between design goals for different XAI user groups and their evaluation methods. From our findings, we develop a framework with step-by-step design guidelines paired with evaluation methods to close the iterative design and evaluation cycles in multidisciplinary XAI teams. Further, we provide summarized ready-to-use tables of evaluation methods and recommendations for different goals in XAI research.

computer science, artificial intelligence
Demystifying XAI: Requirements for Understandable XAI Explanations

Jan Stodt,Christoph Reich,Martin Knahl

DOI: https://doi.org/10.3233/SHTI240477

2024-08-22

Abstract:This paper establishes requirements for assessing the usability of Explainable Artificial Intelligence (XAI) methods, focusing on non-AI experts like healthcare professionals. Through a synthesis of literature and empirical findings, it emphasizes achieving optimal cognitive load, task performance, and task time in XAI explanations. Key components include tailoring explanations to user expertise, integrating domain knowledge, and using non-propositional representations for comprehension. The paper highlights the critical role of relevance, accuracy, and truthfulness in fostering user trust. Practical guidelines are provided for designing transparent and user-friendly XAI explanations, especially in high-stakes contexts like healthcare. Overall, the paper's primary contribution lies in delineating clear requirements for effective XAI explanations, facilitating human-AI collaboration across diverse domains.
A Unified Framework for Evaluating the Effectiveness and Enhancing the Transparency of Explainable AI Methods in Real-World Applications

Md. Ariful Islam,M. F. Mridha,Md Abrar Jahin,Nilanjan Dey

2024-12-05

Abstract:The rapid advancement of deep learning has resulted in substantial advancements in AI-driven applications; however, the "black box" characteristic of these models frequently constrains their interpretability, transparency, and reliability. Explainable artificial intelligence (XAI) seeks to elucidate AI decision-making processes, guaranteeing that explanations faithfully represent the model's rationale and correspond with human comprehension. Despite comprehensive research in XAI, a significant gap persists in standardized procedures for assessing the efficacy and transparency of XAI techniques across many real-world applications. This study presents a unified XAI evaluation framework incorporating extensive quantitative and qualitative criteria to systematically evaluate the correctness, interpretability, robustness, fairness, and completeness of explanations generated by AI models. The framework prioritizes user-centric and domain-specific adaptations, hence improving the usability and reliability of AI models in essential domains. To address deficiencies in existing evaluation processes, we suggest defined benchmarks and a systematic evaluation pipeline that includes data loading, explanation development, and thorough method assessment. The suggested framework's relevance and variety are evidenced by case studies in healthcare, finance, agriculture, and autonomous systems. These provide a solid basis for the equitable and dependable assessment of XAI methodologies. This paradigm enhances XAI research by offering a systematic, flexible, and pragmatic method to guarantee transparency and accountability in AI systems across many real-world contexts.

Artificial Intelligence
Explainable AI (XAI): Core Ideas, Techniques, and Solutions

Rudresh Dwivedi,Devam Dave,Het Naik,Smiti Singhal,Rana Omer,Pankesh Patel,Bin Qian,Zhenyu Wen,Tejal Shah,Graham Morgan,Rajiv Ranjan

DOI: https://doi.org/10.1145/3561048

IF: 16.6

2023-01-18

ACM Computing Surveys

Abstract:As our dependence on intelligent machines continues to grow, so does the demand for more transparent and interpretable models. In addition, the ability to explain the model generally is now the gold standard for building trust and deployment of artificial intelligence systems in critical domains. Explainable artificial intelligence (XAI) aims to provide a suite of machine learning techniques that enable human users to understand, appropriately trust, and produce more explainable models. Selecting an appropriate approach for building an XAI-enabled application requires a clear understanding of the core ideas within XAI and the associated programming frameworks. We survey state-of-the-art programming techniques for XAI and present the different phases of XAI in a typical machine learning development process. We classify the various XAI approaches and, using this taxonomy, discuss the key differences among the existing XAI techniques. Furthermore, concrete examples are used to describe these techniques that are mapped to programming frameworks and software toolkits. It is the intention that this survey will help stakeholders in selecting the appropriate approaches, programming frameworks, and software toolkits by comparing them through the lens of the presented taxonomy.

computer science, theory & methods
Explainable Artificial Intelligence: Evaluating the Objective and Subjective Impacts of xAI on Human-Agent Interaction

Andrew Silva,Mariah Schrum,Erin Hedlund-Botti,Nakul Gopalan,Matthew Gombolay

DOI: https://doi.org/10.1080/10447318.2022.2101698

2022-08-09

Abstract:Intelligent agents must be able to communicate intentions and explain their decision-making processes to build trust, foster confidence, and improve human-agent team dynamics. Recognizing this need, academia and industry are rapidly proposing new ideas, methods, and frameworks to aid in the design of more explainable AI. Yet, there remains no standardized metric or experimental protocol for benchmarking new methods, leaving researchers to rely on their own intuition or ad hoc methods for assessing new concepts. In this work, we present the first comprehensive ( n = 286) user study testing a wide range of approaches for explainable machine learning, including feature importance, probability scores, decision trees, counterfactual reasoning, natural language explanations, and case-based reasoning, as well as a baseline condition with no explanations. We provide the first large-scale empirical evidence of the effects of explainability on human-agent teaming. Our results will help to guide the future of explainability research by highlighting the benefits of counterfactual explanations and the shortcomings of confidence scores for explainability. We also propose a novel questionnaire to measure explainability with human participants, inspired by relevant prior work and correlated with human-agent teaming metrics.
What Does Evaluation of Explainable Artificial Intelligence Actually Tell Us? A Case for Compositional and Contextual Validation of XAI Building Blocks

Kacper Sokol,Julia E. Vogt

DOI: https://doi.org/10.1145/3613905.3651047

2024-03-19

Abstract:Despite significant progress, evaluation of explainable artificial intelligence remains elusive and challenging. In this paper we propose a fine-grained validation framework that is not overly reliant on any one facet of these sociotechnical systems, and that recognises their inherent modular structure: technical building blocks, user-facing explanatory artefacts and social communication protocols. While we concur that user studies are invaluable in assessing the quality and effectiveness of explanation presentation and delivery strategies from the explainees' perspective in a particular deployment context, the underlying explanation generation mechanisms require a separate, predominantly algorithmic validation strategy that accounts for the technical and human-centred desiderata of their (numerical) outputs. Such a comprehensive sociotechnical utility-based evaluation framework could allow to systematically reason about the properties and downstream influence of different building blocks from which explainable artificial intelligence systems are composed -- accounting for a diverse range of their engineering and social aspects -- in view of the anticipated use case.

Human-Computer Interaction,Artificial Intelligence
A review of evaluation approaches for explainable AI with applications in cardiology

Ahmed M Salih,Ilaria Boscolo Galazzo,Polyxeni Gkontra,Elisa Rauseo,Aaron Mark Lee,Karim Lekadir,Petia Radeva,Steffen E Petersen,Gloria Menegaz

DOI: https://doi.org/10.1007/s10462-024-10852-w

Abstract:Explainable artificial intelligence (XAI) elucidates the decision-making process of complex AI models and is important in building trust in model predictions. XAI explanations themselves require evaluation as to accuracy and reasonableness and in the context of use of the underlying AI model. This review details the evaluation of XAI in cardiac AI applications and has found that, of the studies examined, 37% evaluated XAI quality using literature results, 11% used clinicians as domain-experts, 11% used proxies or statistical analysis, with the remaining 43% not assessing the XAI used at all. We aim to inspire additional studies within healthcare, urging researchers not only to apply XAI methods but to systematically assess the resulting explanations, as a step towards developing trustworthy and safe models. Supplementary information: The online version contains supplementary material available at 10.1007/s10462-024-10852-w.
Survey on Explainable AI: From Approaches, Limitations and Applications Aspects

Wenli Yang,Yuchen Wei,Hanyu Wei,Yanyu Chen,Guan Huang,Xiang Li,Renjie Li,Naimeng Yao,Xinyi Wang,Xiaotong Gu,Muhammad Bilal Amin,Byeong Kang

DOI: https://doi.org/10.1007/s44230-023-00038-y

2023-08-11

Human-Centric Intelligent Systems

Abstract:In recent years, artificial intelligence (AI) technology has been used in most if not all domains and has greatly benefited our lives. While AI can accurately extract critical features and valuable information from large amounts of data to help people complete tasks faster, there are growing concerns about the non-transparency of AI in the decision-making process. The emergence of explainable AI (XAI) has allowed humans to better understand and control AI systems, which is motivated to provide transparent explanations for the decisions made by AI. This article aims to present a comprehensive overview of recent research on XAI approaches from three well-defined taxonomies. We offer an in-depth analysis and summary of the status and prospects of XAI applications in several key areas where reliable explanations are urgently needed to avoid mistakes in decision-making. We conclude by discussing XAI's limitations and future research directions.
Explainable AI (XAI): History, Basic Ideas and Methods

Prachi Zodage,Hussain Harianawala,Hafsa Shaikh,Asad Kharodia

DOI: https://doi.org/10.48175/ijarsct-16988

2024-04-07

Abstract:Explainable Artificial Intelligence (XAI) is a field that aims to make artificial intelligence (AI) processes more transparent, explainable, and understandable. As AI processes become more complex, the need to reveal the "black box" nature of these models and provide explanations for their results and decisions is increasing. XAI aims to bridge the gap between the opaque inner workings of AI and human understanding by creating AI that is accurate, useful, and can explain reasoning and decision-making processes in ways that humans can understand. XAI's importance stems from several factors. First, it addresses trust and accountability issues in AI systems, particularly in high-risk sectors like healthcare, finance, and technology. By providing explanations for AI decisions, stakeholders can better understand the logic behind them, detect inconsistencies, and ensure moral and administrative compliance. Second, XAI encourages collaboration and decision-making between people and intelligence, allowing experts and decision-makers to use their knowledge and experience to make better decisions. Thirdly, XAI plays a crucial role in modeling, debugging, and continuous improvement by identifying flaws, biases, or inconsistencies and working to improve performance standards and reliability. Various methods and techniques are used in XAI, each with their own advantages and limitations. Model-free explanations such as LIME, Anchor and SHAP are particularly important because they can be applied to any AI model, regardless of its design or complexity
Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI

Alejandro Barredo Arrieta,Natalia Díaz-Rodríguez,Javier Del Ser,Adrien Bennetot,Siham Tabik,Alberto Barbado,Salvador García,Sergio Gil-López,Daniel Molina,Richard Benjamins,Raja Chatila,Francisco Herrera

2019-12-26

Abstract:In the last years, Artificial Intelligence (AI) has achieved a notable momentum that may deliver the best of expectations over many application sectors across the field. For this to occur, the entire community stands in front of the barrier of explainability, an inherent problem of AI techniques brought by sub-symbolism (e.g. ensembles or Deep Neural Networks) that were not present in the last hype of AI. Paradigms underlying this problem fall within the so-called eXplainable AI (XAI) field, which is acknowledged as a crucial feature for the practical deployment of AI models. This overview examines the existing literature in the field of XAI, including a prospect toward what is yet to be reached. We summarize previous efforts to define explainability in Machine Learning, establishing a novel definition that covers prior conceptual propositions with a major focus on the audience for which explainability is sought. We then propose and discuss about a taxonomy of recent contributions related to the explainability of different Machine Learning models, including those aimed at Deep Learning methods for which a second taxonomy is built. This literature analysis serves as the background for a series of challenges faced by XAI, such as the crossroads between data fusion and explainability. Our prospects lead toward the concept of Responsible Artificial Intelligence, namely, a methodology for the large-scale implementation of AI methods in real organizations with fairness, model explainability and accountability at its core. Our ultimate goal is to provide newcomers to XAI with a reference material in order to stimulate future research advances, but also to encourage experts and professionals from other disciplines to embrace the benefits of AI in their activity sectors, without any prior bias for its lack of interpretability.

Artificial Intelligence,Machine Learning,Neural and Evolutionary Computing
EvaluateXAI: A Framework to Evaluate the Reliability and Consistency of Rule-based XAI Techniques for Software Analytics Tasks

Md Abdul Awal,Chanchal K. Roy

2024-07-19

Abstract:The advancement of machine learning (ML) models has led to the development of ML-based approaches to improve numerous software engineering tasks in software maintenance and evolution. Nevertheless, research indicates that despite their potential successes, ML models may not be employed in real-world scenarios because they often remain a black box to practitioners, lacking explainability in their reasoning. Recently, various rule-based model-agnostic Explainable AI (XAI) techniques, such as PyExplainer and LIME, have been employed to explain the predictions of ML models in software analytics tasks. This paper assesses the ability of these techniques (e.g., PyExplainer and LIME) to generate reliable and consistent explanations for ML models across various software analytics tasks, including Just-in-Time (JIT) defect prediction, clone detection, and the classification of useful code review comments. Our manual investigations find inconsistencies and anomalies in the explanations generated by these techniques. Therefore, we design a novel framework: Evaluation of Explainable AI (EvaluateXAI), along with granular-level evaluation metrics, to automatically assess the effectiveness of rule-based XAI techniques in generating reliable and consistent explanations for ML models in software analytics tasks. After conducting in-depth experiments involving seven state-of-the-art ML models trained on five datasets and six evaluation metrics, we find that none of the evaluation metrics reached 100\%, indicating the unreliability of the explanations generated by XAI techniques. Additionally, PyExplainer and LIME failed to provide consistent explanations for 86.11% and 77.78% of the experimental combinations, respectively. Therefore, our experimental findings emphasize the necessity for further research in XAI to produce reliable and consistent explanations for ML models in software analytics tasks.

Software Engineering

A Trustworthy View on Explainable Artificial Intelligence Method Evaluation.

Exploring Evaluation Methodologies for Explainable AI: Guidelines for Objective and Subjective Assessment

Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial Intelligence

A New Perspective on Evaluation Methods for Explainable Artificial Intelligence (XAI)

The Analysis and Development of an XAI Process on Feature Contribution Explanation.

EXplainable Artificial Intelligence (XAI)—From Theory to Methods and Applications

Study on the Helpfulness of Explainable Artificial Intelligence

Explainable artificial intelligence: a comprehensive review

Metrics for Explainable AI: Challenges and Prospects

Explainability of Artificial Intelligence Methods, Applications and Challenges: A Comprehensive Survey

A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems

Demystifying XAI: Requirements for Understandable XAI Explanations

A Unified Framework for Evaluating the Effectiveness and Enhancing the Transparency of Explainable AI Methods in Real-World Applications

Explainable AI (XAI): Core Ideas, Techniques, and Solutions

Explainable Artificial Intelligence: Evaluating the Objective and Subjective Impacts of xAI on Human-Agent Interaction

What Does Evaluation of Explainable Artificial Intelligence Actually Tell Us? A Case for Compositional and Contextual Validation of XAI Building Blocks

A review of evaluation approaches for explainable AI with applications in cardiology

Survey on Explainable AI: From Approaches, Limitations and Applications Aspects

Explainable AI (XAI): History, Basic Ideas and Methods

Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI

EvaluateXAI: A Framework to Evaluate the Reliability and Consistency of Rule-based XAI Techniques for Software Analytics Tasks