Abstract:The success of current deep saliency models heavily depends on large amounts of annotated human fixation data to fit the highly non-linear mapping between the stimuli and visual saliency. Such fully supervised data-driven approaches are annotation-intensive and often fail to consider the underlying mechanisms of visual attention. In contrast, in this paper, we introduce a model based on various cognitive theories of visual saliency, which learns visual attention patterns in a weakly supervised manner. Our approach incorporates insights from cognitive science as differentiable submodules, resulting in a unified, end-to-end trainable framework. Specifically, our model encapsulates the following important components motivated from biological vision. (a) As scene semantics are closely related to visually attentive regions, our model encodes discriminative spatial information for scene understanding through spatial visual semantics embedding. (b) To model the objectness factors in visual attention deployment, we incorporate object-level semantics embedding and object relation information. (c) Considering the "winner-take-all" mechanism in visual stimuli processing, we model the competition mechanism among objects with softmax based neural attention. (d) Lastly, a conditional center prior is learned to mimic the spatial distribution bias of visual attention. Furthermore, we propose novel loss functions to utilize supervision cues from image-level semantics, saliency prior knowledge, and self-information compression. Experiments show that our method achieves promising results, and even outperforms many of its fully supervised counterparts. Overall, our weakly supervised saliency method makes an essential step towards reducing the annotation budget of current approaches, as well as providing a more comprehensive understanding of the visual attention mechanism. Our code is available at: https://github.com/ashleylqx/WeakFixation.git.

Noise-Aware Video Saliency Prediction

Deep Learning for Video Saliency Detection.

Revisiting Video Saliency Prediction in the Deep Learning Era

AIM 2024 Challenge on Video Saliency Prediction: Methods and Results

Predicting Video Saliency Using Crowdsourced Mouse-Tracking Data

SAL3D: a model for saliency prediction in 3D meshes

NPF-200: A Multi-Modal Eye Fixation Dataset and Method for Non-Photorealistic Videos

What Do Deep Saliency Models Learn about Visual Attention?

A Gated Fusion Network for Dynamic Saliency Prediction

ViDaS Video Depth-aware Saliency Network

Uncertainty Inspired RGB-D Saliency Detection

CASP-Net: Rethinking Video Saliency Prediction from an Audio-VisualConsistency Perceptual Perspective

Saliency Detection in Educational Videos: Analyzing the Performance of Current Models, Identifying Limitations and Advancement Directions

An event-based implementation of saliency-based visual attention for rapid scene analysis

Video Saliency Detection via Dynamic Consistent Spatio-Temporal Attention Modelling.

Weakly Supervised Visual Saliency Prediction

GASP: Gated Attention For Saliency Prediction

Saliency Prediction in the Deep Learning Era: Successes and Limitations

A Dataset and Evaluation Methodology for Visual Saliency in Video

End-to-End Video Saliency Detection Via a Deep Contextual Spatiotemporal Network

Unsupervised uncertainty estimation using spatiotemporal cues in video saliency detection