Abstract:The rapid development of deep learning has greatly improved the performance of various computer vision tasks. However, the "black box" nature of deep learning network models makes it difficult for users to understand its decision-making mechanism, which is not conductive to model structure optimization and security enhancement and also greatly increases the training cost. Focusing on the task of intelligent image interpretation, this paper makes a comprehensive review and comparison of the research progress of deep learning interpretability. Firstly, we group the current interpretability analysis methods into six categories: activation maximization method, surrogate model, attribution method, perturbation-based method, class activation map based method and example-based method, and review the principle, focus, advantages, and disadvantages of existing related works. Secondly, we introduce eight evaluation metrics that measure the reliability of the explanations provided by the various interpretability analysis methods, and sort out the current publicly available open source libraries for deep learning interpretability analysis. Based on the open source library, we verify the applicability of the current deep learning interpretability analysis methods to the interpretation of remote sensing images. The experimental results show that the current interpretability methods are applicable to the analysis of remote sensing interpretation, but have certain limitations. Finally, we summarize the open challenges of using existing interpretability algorithms for remote sensing data analysis, and look forward to the prospect of designing interpretability analysis methods oriented to remote sensing images. We hope this review can promote the research on interpretability methods for remote sensing image interpretation, so as to provide reliable theoretical support and algorithm design guidance for the application of deep learning technology in remote sensing image interpretation tasks.

A User-Based Taxonomy for Deep Learning Visualization

Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers

DeepEye: Towards Automatic Data Visualization

Diverse Interaction Recommendation for Public Users Exploring Multi-view Visualization Using Deep Learning

Visualization Assessment - A Machine Learning Approach.

Understanding Neural Networks Through Deep Visualization

NeuralVis: Visualizing and Interpreting Deep Learning Models

A Comprehensive Review of Deep Neural Network Interpretation Using Topological Data Analysis

Visual Analytics for Explainable Deep Learning

Explaining Deep Face Algorithms through Visualization: A Survey

Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models

Feature Selection on Deep Learning Models: an Interactive Visualization Approach

Deep Learning Interpretability Analysis Methods in Image Interpretation

Dissecting Deep Learning Networks—Visualizing Mutual Information

Visualizing dataflow graphs of deep learning models in tensorflow

A survey of visual analytics techniques for machine learning

A Survey of Explainable AI in Deep Visual Modeling: Methods and Metrics

ConceptExplainer: Interactive Explanation for Deep Neural Networks from a Concept Perspective

A Survey on Explainable Artificial Intelligence (XAI) Techniques for Visualizing Deep Learning Models in Medical Imaging

Visual Interpretability for Deep Learning: a Survey

DeepVisual: a visual programming tool for deep learning systems