Abstract:Data size is the bottleneck for developing deep saliency models, because collecting eye-movement data is very time-consuming and expensive. Most of current studies on human attention and saliency modeling have used high-quality stereotype stimuli. In real world, however, captured images undergo various types of transformations. Can we use these transformations to augment existing saliency datasets? Here, we first create a novel saliency dataset including fixations of 10 observers over 1900 images degraded by 19 types of transformations. Second, by analyzing eye movements, we find that observers look at different locations over transformed versus original images. Third, we utilize the new data over transformed images, called data augmentation transformation (DAT), to train deep saliency models. We find that label-preserving DATs with negligible impact on human gaze boost saliency prediction, whereas some other DATs that severely impact human gaze degrade the performance. These label-preserving valid augmentation transformations provide a solution to enlarge existing saliency datasets. Finally, we introduce a novel saliency model based on generative adversarial networks (dubbed GazeGAN). A modified U-Net is utilized as the generator of the GazeGAN, which combines classic “skip connection” with a novel “center-surround connection” (CSC) module. Our proposed CSC module mitigates trivial artifacts while emphasizing semantic salient regions, and increases model nonlinearity, thus demonstrating better robustness against transformations. Extensive experiments and comparisons indicate that GazeGAN achieves state-of-the-art performance over multiple datasets. We also provide a comprehensive comparison of 22 saliency models on various transformed scenes, which contributes a new robustness benchmark to saliency community. Our code and dataset are available at: https://github.com/CZHQuality/Sal-CFS-GAN.

Influence of Spatial Resolution on State-of-the-Art Saliency Models.

A Survey of Visual Attention and Saliency Computation

Saliency Revisited: Analysis of Mouse Movements versus Fixations

Learning visual saliency based on object's relative relationship

What Do Deep Saliency Models Learn about Visual Attention?

Deep saliency models learn low-, mid-, and high-level features to predict scene attention

Toward Statistical Modeling of Saccadic Eye-Movement and Visual Saliency.

Learning to Model Task-Oriented Attention

Review of Visual Saliency Detection with Comprehensive Information

Invariance Analysis of Saliency Models Versus Human Gaze During Scene Free Viewing

Saliency Preservation in Low-Resolution Grayscale Images

Saliency detection using a central stimuli sensitivity based model

Saliency Prediction in the Deep Learning Era: Successes and Limitations

Saliency In Crowd

Predicting eye fixations with higher-level visual features.

A Psychophysically Oriented Saliency Map Prediction Model

Biological Plausibility Of Spectral Domain Approach For Spatiotemporal Visual Saliency

Deep supervised visual saliency model addressing low-level features

A Deep Model of Visual Attention for Saliency Detection on 3D Objects

How is Gaze Influenced by Image Transformations? Dataset and Model

Low-level and High-Level Prior Learning for Visual Saliency Estimation.