Abstract:This paper studies audio-visual deep saliency prediction. It introduces a conceptually simple and effective Deep Audio-Visual Embedding for dynamic saliency prediction dubbed ``DAVE" in conjunction with our efforts towards building an Audio-Visual Eye-tracking corpus named ``AVE". Despite existing a strong relation between auditory and visual cues for guiding gaze during perception, video saliency models only consider visual cues and neglect the auditory information that is ubiquitous in dynamic scenes. Here, we investigate the applicability of audio cues in conjunction with visual ones in predicting saliency maps using deep neural networks. To this end, the proposed model is intentionally designed to be simple. Two baseline models are developed on the same architecture which consists of an encoder-decoder. The encoder projects the input into a feature space followed by a decoder that infers saliency. We conduct an extensive analysis on different modalities and various aspects of multi-model dynamic saliency prediction. Our results suggest that (1) audio is a strong contributing cue for saliency prediction, (2) salient visible sound-source is the natural cause of the superiority of our Audio-Visual model, (3) richer feature representations for the input space leads to more powerful predictions even in absence of more sophisticated saliency decoders, and (4) Audio-Visual model improves over 53.54\% of the frames predicted by the best Visual model (our baseline). Our endeavour demonstrates that audio is an important cue that boosts dynamic video saliency prediction and helps models to approach human performance. The code is available at <a class="link-external link-https" href="https://github.com/hrtavakoli/DAVE" rel="external noopener nofollow">this https URL</a>

A Novel Lightweight Audio-visual Saliency Model for Videos

Lavs - A Lightweight Audio-Visual Saliency Prediction Model.

Unified Audio-Visual Saliency Model for Omnidirectional Videos with Spatial Audio

A Multimodal Saliency Model For Videos With High Audio-Visual Correspondence

Audio-visual Aligned Saliency Model for Omnidirectional Video with Implicit Neural Representation Learning

Learning to Predict Salient Faces: A Novel Visual-Audio Saliency Model

Audiovisual Saliency Prediction Via Deep Learning

From Discrete Representation to Continuous Modeling: A Novel Audio-Visual Saliency Prediction Model with Implicit Neural Representations

Audio-visual Saliency Prediction Model with Implicit Neural Representation

Deep Audio-Visual Fusion Neural Network for Saliency Estimation.

A Comprehensive Survey on Video Saliency Detection with Auditory Information: the Audio-visual Consistency Perceptual is the Key!

Relevance-guided Audio Visual Fusion for Video Saliency Prediction

Towards Audio-Visual Saliency Prediction for Omnidirectional Video with Spatial Audio

Video Saliency Detection Using Deep Convolutional Neural Networks.

MTCAM: A Novel Weakly-Supervised Audio-Visual Saliency Prediction Model with Multi-Modal Transformer

SVGC-AVA: 360-Degree Video Saliency Prediction with Spherical Vector-Based Graph Convolution and Audio-Visual Attention

DAVE: A Deep Audio-Visual Embedding for Dynamic Saliency Prediction

DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction

An Attention Based Speaker-Independent Audio-Visual Deep Learning Model for Speech Enhancement

Audiovisual Saliency Prediction in Uncategorized Video Sequences based on Audio-Video Correlation

How Does Audio Influence Visual Attention in Omnidirectional Videos? Database and Model