Abstract:Machine learning-based systems have presented increasing learning performance, in a wide variety of tasks. However, the problem with some state-of-the-art models is their lack of transparency, trustworthiness, and explainability. To address this problem, eXplainable Artificial Intelligence (XAI) appeared. It is a research field that aims to make black-box models more understandable to humans. The research on this topic has increased in recent years, and many methods, such as LIME (Local Interpretable Model-Agnostic Explanations) and SHAP (SHapley Additive exPlanations) have been proposed. Machine learning-based Intrusion Detection Systems (IDS) are one of the many application domains of XAI. However, most of the works about model interpretation focus on other fields, like computer vision, natural language processing, biology, healthcare, etc. This poses a challenge for cybersecurity professionals tasked with analyzing IDS results, thereby impeding their capacity to make informed decisions. In an attempt to address this problem, we have selected two XAI methods, LIME, and SHAP. Using the methods, we have retrieved explanations for the results of a black-box model, part of an IDS solution that performs intrusion detection on IoT devices, increasing its interpretability. In order to validate the explanations, we carried out a perturbation analysis where we tried to obtain a different classification based on the features present in the explanations. With the explanations and the perturbation analysis we were able to draw conclusions about the negative impact of particular features on the model results when present in the input data, making it easier for cybersecurity experts when analyzing the model results and it serves as an aid to the continuous improvement the model. The perturbations also serve as a comparison of performance between LIME and SHAP. To evaluate the degree of interpretability increase, and the explanations provided by each XAI method of the model and directly compare the XAI methods, we have performed a survey analysis.

Explainable AI: Using Shapley Value to Explain Complex Anomaly Detection ML-Based Systems

Shapley Value: from Cooperative Game to Explainable Artificial Intelligence

Explainable AI: Efficiency Sequential Shapley Updating Approach

Error Analysis of Shapley Value-Based Model Explanations: An Informative Perspective

Explainable AI for Intrusion Detection Systems: LIME and SHAP Applicability on Multi-Layer Perceptron

Local Interpretable Model Agnostic Shap Explanations for machine learning models

Collective eXplainable AI: Explaining Cooperative Strategies and Agent Contribution in Multiagent Reinforcement Learning with Shapley Values

Shapley-based Explainable AI for Clustering Applications in Fault Diagnosis and Prognosis

Explaining a Machine-Learning Lane Change Model With Maximum Entropy Shapley Values

Explainable Anomaly Detection Framework for Maritime Main Engine Sensor Data

Explainable AI in Credit Risk Management

Fooling SHAP with Output Shuffling Attacks

Using Kernel SHAP XAI Method to optimize the Network Anomaly Detection Model

Utilizing XAI Technique to Improve Autoencoder based Model for Computer Network Anomaly Detection with Shapley Additive Explanation(SHAP)

Explainable AI for survival analysis: a median-SHAP approach

On the Failings of Shapley Values for Explainability

Explaining Predictive Uncertainty with Information Theoretic Shapley Values

CAGE: Causality-Aware Shapley Value for Global Explanations

Separating Prediction and Explanation: An Approach Based on Explainable Artificial Intelligence for Analyzing Network Intrusion

Manifold-based Shapley explanations for high dimensional correlated features

Abnormality Detection and Failure Prediction Using Explainable Bayesian Deep Learning: Methodology and Case Study with Industrial Data