Abstract:In virtual reality (VR), correct and precise estimations of user’s visual fixations and head movements can enhance the quality of experience by allocating more computation resources for analysing and rendering on the areas of interest. However, there is insufficient research about understanding the visual exploration of users when modeling VR visual attention. To bridge the gap between the saliency prediction for traditional 2D content and omnidirectional content, we construct the visual attention dataset and propose the visual saliency prediction framework for panoramic videos. Around the instantaneous viewing behavior, we propose a traditional method to adapt 2D saliency models and design a CNN-based model to better predict visual saliency. In the proposed traditional model, mechanism of visual attention and viewing behaviors are considered in the computation of edge weights on graphs which are interpreted as Markov chains. The fraction of the visual attention that is diverted to each high-clarity vision (HCV) area is estimated through equilibrium distribution of this chain. We also propose the Graph-Based CNN model. The RGB channel and optical flow form the spatial-temporal units of HCVs, from which node feature vectors are extracted. Graph convolution is used to learn the mutual information between node feature vectors of HCVs and retain geometric information. Then feature vectors are aligned according to geometry structure of equirectangular format, and the feature decoder maps the aligned feature maps to the data distribution. We also construct the dynamic omnidirectional monocular (DOM) saliency dataset with 64 diverse videos evaluated by 28 people. The subjective results show that the instantaneous viewing behavior is important in the VR experience. Extensive experiments are conducted on the dataset and the results demonstrate the effectiveness of the proposed framework. The dataset will be released to facilitate the future studies related to visual saliency prediction for 360-degree contents.

SalGFCN: Graph Based Fully Convolutional Network for Panoramic Saliency Prediction

Dilated Convolutional Neural Networks for Panoramic Image Saliency Prediction

SalGCN: Saliency Prediction for 360-Degree Images Based on Spherical Graph Convolutional Networks

Saliency Prediction Network for $360^\circ$ Videos

Multi-scale graph feature extraction network for panoramic image saliency detection

Intra- and Inter-Reasoning Graph Convolutional Network for Saliency Prediction on 360° Images

Spherical Convolution-based Saliency Detection for FoV Prediction in 360-degree Video Streaming

360Spred: Saliency Prediction for 360-Degree Videos Based on 3D Separable Graph Convolutional Networks

SalBiNet360: Saliency Prediction on 360° Images with Local-Global Bifurcated Deep Network

MRGAN360: Multi-stage Recurrent Generative Adversarial Network for 360 Degree Image Saliency Prediction

Viewing Behavior Supported Visual Saliency Predictor for 360 Degree Videos

360 Degree Panorama Synthesis from Sequential Views Based on Improved FC-Densenets

Viewport Prediction for Panoramic Video with Multi-CNN

Attentive and Context-Aware Deep Network for Saliency Prediction on Omni-Directional Images

Multi-scale transformer network for saliency prediction on 360-degree images

Cross-Modality Fusion and Progressive Integration Network for Saliency Prediction on Stereoscopic 3D Images

Panoramic Vision Transformer for Saliency Detection in 360° Videos

A Multi-FoV Viewport-Based Visual Saliency Model Using Adaptive Weighting Losses for 360$^\circ$ Images

Viewport-Dependent Saliency Prediction in 360 Video

A New Deep Distortion Convolutional Neural Network for Semantic Segmentation of Panoramic Images

FGO-Net: Feature and Gaussian Optimization Network for visual saliency prediction