Abstract:The rapid development of deep learning has greatly improved the performance of various computer vision tasks. However, the "black box" nature of deep learning network models makes it difficult for users to understand its decision-making mechanism, which is not conductive to model structure optimization and security enhancement and also greatly increases the training cost. Focusing on the task of intelligent image interpretation, this paper makes a comprehensive review and comparison of the research progress of deep learning interpretability. Firstly, we group the current interpretability analysis methods into six categories: activation maximization method, surrogate model, attribution method, perturbation-based method, class activation map based method and example-based method, and review the principle, focus, advantages, and disadvantages of existing related works. Secondly, we introduce eight evaluation metrics that measure the reliability of the explanations provided by the various interpretability analysis methods, and sort out the current publicly available open source libraries for deep learning interpretability analysis. Based on the open source library, we verify the applicability of the current deep learning interpretability analysis methods to the interpretation of remote sensing images. The experimental results show that the current interpretability methods are applicable to the analysis of remote sensing interpretation, but have certain limitations. Finally, we summarize the open challenges of using existing interpretability algorithms for remote sensing data analysis, and look forward to the prospect of designing interpretability analysis methods oriented to remote sensing images. We hope this review can promote the research on interpretability methods for remote sensing image interpretation, so as to provide reliable theoretical support and algorithm design guidance for the application of deep learning technology in remote sensing image interpretation tasks.

A Feature Structure Based Interpretability Evaluation Approach for Deep Learning

Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond

Relevance Inference Based on Direct Contribution: Counterfactual Explanation to Deep Networks for Intelligent Decision-making

Interpretable Artificial Intelligence through the Lens of Feature Interaction

How far from automatically interpreting deep learning

A Survey of the Interpretability Aspect of Deep Learning Models

Deep Learning Interpretability Analysis Methods in Image Interpretation

Deep PLS: A Lightweight Deep Learning Model for Interpretable and Efficient Data Analytics

Interpretability of deep learning models: A survey of results

Towards Understanding Sensitive and Decisive Patterns in Explainable AI: A Case Study of Model Interpretation in Geometric Deep Learning

Feature Analysis Network: An Interpretable Idea in Deep Learning

FENet: A Feature Explanation Network with a Hierarchical Interpretable Architecture for Intelligent Decision-Making

Interpreting Deep Learning Model Using Rule-based Method

Interpreting Deep Neural Networks Through Model Transformation: Literature Review

Interpretability for Reliable, Efficient, and Self-Cognitive DNNs: from Theories to Applications.

Interpreting Deep Learning Models in Natural Language Processing: A Review

Interpretable Deep Convolutional Neural Networks via Meta-learning

M-Rule: an Enhanced Deep Taylor Decomposition for Multi-model Interpretability

On Interpretability of Artificial Neural Networks: A Survey

Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks

Explaining the Unexplained: Revealing Hidden Correlations for Better Interpretability