Exploring Viewport Features for Semi-Supervised Saliency Prediction in Omnidirectional Images

Mengke Huang,Gongyang Li,Zhi Liu,Yong Wu,Chen Gong,Linchao Zhu,Yi Yang

DOI: https://doi.org/10.1016/j.imavis.2022.104590

IF: 3.86

2023-01-01

Image and Vision Computing

Abstract:Compared with the annotated data for the 2D image saliency prediction task, the annotated data for training om-nidirectional image (or 360 degrees image) saliency prediction models are not sufficient. Most existing fully-supervised saliency prediction methods for omnidirectional images (ODIs) adopt a scheme, first training the methods on a labeled large 2D image saliency prediction dataset and then fine-tuning the methods on the labeled tiny ODI sa-liency prediction dataset. However, this strategy is time-consuming and may not inadequately mine the visual features built in ODIs. To explore the visual attributes targeted at ODIs and address the shortage of labels on these ODIs, in this paper, we propose an end-to-end semi-supervised network, namely VFNet, which relies on viewport features and only utilizes ODIs as training data, for ODI saliency prediction. Concretely, we adopt con-sistency regularization as our semi-supervised learning framework. The predictions between main and auxiliary saliency inference networks in the VFNet enforce consistency. Aiming at ODIs, we introduce a new form of per-turbation, i.e., DropView, to improve the effectiveness of consistency regularization. By randomly dropping out different 360 degrees cubemap viewport features before the auxiliary saliency inference network, the proposed DropView enhances the robustness of the final ODI saliency prediction. To adaptively interact with the equirectangular and different cubemap viewport features according to their contributions, we introduce a Viewport Feature Adaptive Integration (VFAI) module and deploy the VFAI module at different levels in the VFNet to raise the capacity of feature encoding of our VFNet. Compared with state-of-the-art fully-supervised methods, our VFNet with fewer labeled training data achieves competitive performance demonstrated by exten-sive experiments on two publicly available datasets. (c) 2022 Elsevier B.V. All rights reserved.