Abstract:Understanding and predicting viewer attention in omnidirectional videos (ODVs) is crucial for enhancing user engagement in virtual and augmented reality applications. Although both audio and visual modalities are essential for saliency prediction in ODVs, the joint exploitation of these two modalities has been limited, primarily due to the absence of large-scale audio-visual saliency databases and comprehensive analyses. This paper comprehensively investigates audio-visual attention in ODVs from both subjective and objective perspectives. Specifically, we first introduce a new audio-visual saliency database for omnidirectional videos, termed AVS-ODV database, containing 162 ODVs and corresponding eye movement data collected from 60 subjects under three audio modes including mute, mono, and ambisonics. Based on the constructed AVS-ODV database, we perform an in-depth analysis of how audio influences visual attention in ODVs. To advance the research on audio-visual saliency prediction for ODVs, we further establish a new benchmark based on the AVS-ODV database by testing numerous state-of-the-art saliency models, including visual-only models and audio-visual models. In addition, given the limitations of current models, we propose an innovative omnidirectional audio-visual saliency prediction network (OmniAVS), which is built based on the U-Net architecture, and hierarchically fuses audio and visual features from the multimodal aligned embedding space. Extensive experimental results demonstrate that the proposed OmniAVS model outperforms other state-of-the-art models on both ODV AVS prediction and traditional AVS predcition tasks. The AVS-ODV database and OmniAVS model will be released to facilitate future research.

Sound Influences Visual Attention Discriminately in Videos

Asymmetrical Cross-Modal Influence on Neural Encoding of Auditory and Visual Features in Natural Scenes.

How Sound Affects Visual Attention in Omnidirectional Videos.

How Does Audio Influence Visual Attention in Omnidirectional Videos? Database and Model

A Multimodal Saliency Model For Videos With High Audio-Visual Correspondence

Unified Audio-Visual Saliency Model for Omnidirectional Videos with Spatial Audio

Audiovisual Integration of Natural Auditory and Visual Stimuli in the Real-World Situation

Fixation prediction through multimodal analysis

Locating the Sources for Cross-Modal Interactions and Decision Making During Judging the Visual-Affected Auditory Intensity Change.

A Comprehensive Survey on Video Saliency Detection with Auditory Information: the Audio-visual Consistency Perceptual is the Key!

The Behavioral Presentations of Audio-Visual Integration in Rats

Audio-visual Aligned Saliency Model for Omnidirectional Video with Implicit Neural Representation Learning

The Influence of Text-guidance on Visual Attention.

A Review of Visual and Auditory Attention Computational Models

What can computational models learn from human selective attention? A review from an audiovisual crossmodal perspective

The Effects of Bilateral and Ipsilateral Auditory Stimuli on the Subcomponents of Visual Attention

A Novel Lightweight Audio-visual Saliency Model for Videos

Visual-aural attention modeling for talk show video highlight detection

The Influence of Selective and Divided Attention on Audiovisual Integration in Children

Audio-Visual Saliency for Omnidirectional Videos

Crossmodal Integration Enhances Neural Representation of Task-Relevant Features in Audiovisual Face Perception