Abstract:By highlighting important features that contribute to model prediction, visual saliency is used as a natural form to interpret the working mechanism of deep neural networks. Numerous methods have been proposed to achieve better saliency results. However, we find that previous visual saliency methods are not reliable enough to provide meaningful interpretation through a simple sanity check: saliency methods are required to explain the output of non-maximum prediction classes, which are usually not ground-truth classes. For example, let the methods interpret an image of "dog" given a wrong class label "fish" as the query. This procedure can test whether these methods reliably interpret model's predictions based on existing features that appear in the data. Our experiments show that previous methods failed to pass the test by generating similar saliency maps or scattered patterns. This false saliency response can be dangerous in certain scenarios, such as medical diagnosis. We find that these failure cases are mainly due to the attribution vanishing and adversarial noise within these methods. In order to learn reliable visual saliency, we propose a simple method that requires the output of the model to be close to the original output while learning an explanatory saliency mask. To enhance the smoothness of the optimized saliency masks, we then propose a simple Hierarchical Attribution Fusion (HAF) technique. In order to fully evaluate the reliability of visual saliency methods, we propose a new task Disturbed Weakly Supervised Object Localization (D-WSOL) to measure whether these methods can correctly attribute the model's output to existing features. Experiments show that previous methods fail to meet this standard, and our approach helps to improve the reliability by suppressing false saliency responses. After observing a significant layout difference in saliency masks between real and adversarial samples. we propose to train a simple CNN on these learned hierarchical attribution masks to distinguish adversarial samples. Experiments show that our method can improve detection performance over other approaches significantly.

Domain Separation Graph Neural Networks for Saliency Object Ranking

Salient Object Ranking with Position-Preserved Attention

Instance-Level Relative Saliency Ranking with Graph Reasoning

Bi-directional Object-Context Prioritization Learning for Saliency Ranking

HyperSOR: Context-aware Graph Hypernetwork for Salient Object Ranking

SGR: An Improved Point-Based Method for Remote Sensing Object Detection via Dual-Domain Alignment Saliency-Guided RepPoints

Fine-Grained Image Classification Via Spatial Saliency Extraction.

Global–Local Semantic Interaction Network for Salient Object Detection in Optical Remote Sensing Images With Scribble Supervision

Scene Classification of Remote Sensing Images Based on Saliency Dual Attention Residual Network

Transcending Pixels: Boosting Saliency Detection via Scene Understanding from Aerial Imagery

ORSI Salient Object Detection via Progressive Semantic Flow and Uncertainty-Aware Refinement

Inferring Attention Shifts for Salient Instance Ranking

DeepSaliency: Multi-Task Deep Neural Network Model for Salient Object Detection

Object Detection Based on Global-Local Saliency Constraint in Aerial Images

Learning Reliable Visual Saliency for Model Explanations

Salient Object Detection via Bilateral Feature Fusion and Score Sorting Attention Mechanism

SCG: Saliency and Contour Guided Salient Instance Segmentation

Rethinking Object Saliency Ranking: A Novel Whole-flow Processing Paradigm

Rethinking of the Image Salient Object Detection: Object-level Semantic Saliency Re-ranking First, Pixel-wise Saliency Refinement Latter

Spatial-Aware Object-Level Saliency Prediction by Learning Graphlet Hierarchies

Boundary-semantic collaborative guidance network with dual-stream feedback mechanism for salient object detection in optical remote sensing imagery