Abstract:The rapid development of deep learning has greatly improved the performance of various computer vision tasks. However, the "black box" nature of deep learning network models makes it difficult for users to understand its decision-making mechanism, which is not conductive to model structure optimization and security enhancement and also greatly increases the training cost. Focusing on the task of intelligent image interpretation, this paper makes a comprehensive review and comparison of the research progress of deep learning interpretability. Firstly, we group the current interpretability analysis methods into six categories: activation maximization method, surrogate model, attribution method, perturbation-based method, class activation map based method and example-based method, and review the principle, focus, advantages, and disadvantages of existing related works. Secondly, we introduce eight evaluation metrics that measure the reliability of the explanations provided by the various interpretability analysis methods, and sort out the current publicly available open source libraries for deep learning interpretability analysis. Based on the open source library, we verify the applicability of the current deep learning interpretability analysis methods to the interpretation of remote sensing images. The experimental results show that the current interpretability methods are applicable to the analysis of remote sensing interpretation, but have certain limitations. Finally, we summarize the open challenges of using existing interpretability algorithms for remote sensing data analysis, and look forward to the prospect of designing interpretability analysis methods oriented to remote sensing images. We hope this review can promote the research on interpretability methods for remote sensing image interpretation, so as to provide reliable theoretical support and algorithm design guidance for the application of deep learning technology in remote sensing image interpretation tasks.

Interpretability of Machine Learning Methods Applied to Neuroimaging

Applications of interpretable deep learning in neuroimaging: a comprehensive review

Looking deeper into interpretable deep learning in neuroimaging: a comprehensive survey

A Survey of the Interpretability Aspect of Deep Learning Models

Interpretability of deep learning models: A survey of results

Interpreting Brain Biomarkers: Challenges and solutions in interpreting machine learning-based predictive neuroimaging

A Survey on Neural Network Interpretability

Interpretable Deep Learning for Neuroimaging-Based Diagnostic Classification

Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond

How good Neural Networks interpretation methods really are? A quantitative benchmark

Machine Learning Interpretability: A Survey on Methods and Metrics

Explainable AI: A review of applications to neuroimaging data

On Interpretability of Artificial Neural Networks: A Survey

Identifying Interpretable Visual Features in Artificial and Biological Neural Systems

Transparency of deep neural networks for medical image analysis: A review of interpretability methods

Interpreting Deep Learning Models in Natural Language Processing: A Review

Deep Learning Interpretability Analysis Methods in Image Interpretation

Interpretable machine learning for psychological research: Opportunities and pitfalls.

Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications

A psychophysics approach for quantitative comparison of interpretable computer vision models

Interpretability of Machine Learning: Recent Advances and Future Prospects