Abstract:Learning common subspace is prevalent way in cross-modal retrieval to solve the problem of data from different modalities having inconsistent distributions and representations that cannot be directly compared. Previous cross-modal retrieval methods focus on projecting the cross-modal data into a common space by learning the correlation between them to bridge the modality gap. However, the rich semantic information in the video and the heterogeneous nature of audio-visual data leads to more serious heterogeneous gaps intuitively, which may lead to the loss of key semantic content of video with single clue by the previous methods when eliminating the modality gap, while the semantics of the categories may undermine the properties of the original features. In this work, we aim to learn effective audio-visual representations to support audio-visual cross-modal retrieval (AVCMR). We propose a novel model that maps audio-visual modalities into two distinct shared latent subspaces: explicit and implicit shared spaces. In particular, the explicit shared space is used to optimize pairwise correlations, where learned representations across modalities capture the commonalities of audio-visual pairs and reduce the modality gap. The implicit shared space is used to preserve the distinctive features between modalities by maintaining the discrimination of audio/video patterns from different semantic categories. Finally, the fusion of the features learned from the two latent subspaces is used for the similarity computation of the AVCMR task. The comprehensive experimental results on two audio-visual datasets demonstrate that our proposed model for using two different latent subspaces for audio-visual cross-modal learning is effective and significantly outperforms the state-of-the-art cross-modal models that learn features from a single subspace.

Audio-visual Aligned Saliency Model for Omnidirectional Video with Implicit Neural Representation Learning

Unified Audio-Visual Saliency Model for Omnidirectional Videos with Spatial Audio

Audio-visual Saliency Prediction Model with Implicit Neural Representation

From Discrete Representation to Continuous Modeling: A Novel Audio-Visual Saliency Prediction Model with Implicit Neural Representations

How Does Audio Influence Visual Attention in Omnidirectional Videos? Database and Model

Towards Audio-Visual Saliency Prediction for Omnidirectional Video with Spatial Audio

Audio-Visual Saliency for Omnidirectional Videos

A Novel Lightweight Audio-visual Saliency Model for Videos

A Multimodal Saliency Model For Videos With High Audio-Visual Correspondence

How to Teach DNNs to Pay Attention to the Visual Modality in Speech Recognition

Lavs - A Lightweight Audio-Visual Saliency Prediction Model.

Audio-Visual Speech Separation with Visual Features Enhanced by Adversarial Training

Audiovisual Saliency Prediction Via Deep Learning

Self-supervised Learning of Audio Representations from Audio-Visual Data using Spatial Alignment

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

SVGC-AVA: 360-Degree Video Saliency Prediction with Spherical Vector-Based Graph Convolution and Audio-Visual Attention

Audio-visual saliency prediction with multisensory perception and integration

A Comprehensive Survey on Video Saliency Detection with Auditory Information: the Audio-visual Consistency Perceptual is the Key!

Improving Audio-Visual Segmentation with Bidirectional Generation.

Aligning Audio-Visual Joint Representations with an Agentic Workflow

DAVE: A Deep Audio-Visual Embedding for Dynamic Saliency Prediction