Abstract:Learning common subspace is prevalent way in cross-modal retrieval to solve the problem of data from different modalities having inconsistent distributions and representations that cannot be directly compared. Previous cross-modal retrieval methods focus on projecting the cross-modal data into a common space by learning the correlation between them to bridge the modality gap. However, the rich semantic information in the video and the heterogeneous nature of audio-visual data leads to more serious heterogeneous gaps intuitively, which may lead to the loss of key semantic content of video with single clue by the previous methods when eliminating the modality gap, while the semantics of the categories may undermine the properties of the original features. In this work, we aim to learn effective audio-visual representations to support audio-visual cross-modal retrieval (AVCMR). We propose a novel model that maps audio-visual modalities into two distinct shared latent subspaces: explicit and implicit shared spaces. In particular, the explicit shared space is used to optimize pairwise correlations, where learned representations across modalities capture the commonalities of audio-visual pairs and reduce the modality gap. The implicit shared space is used to preserve the distinctive features between modalities by maintaining the discrimination of audio/video patterns from different semantic categories. Finally, the fusion of the features learned from the two latent subspaces is used for the similarity computation of the AVCMR task. The comprehensive experimental results on two audio-visual datasets demonstrate that our proposed model for using two different latent subspaces for audio-visual cross-modal learning is effective and significantly outperforms the state-of-the-art cross-modal models that learn features from a single subspace.

Multi-grained Encoding and Joint Embedding Space Fusion for Video and Text Cross-Modal Retrieval

Fine-Grained Cross-Modal Retrieval with Triple-Streamed Memory Fusion Transformer Encoder

Stacked Convolutional Deep Encoding Network for Video-Text Retrieval.

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Cross-Modal Learning Based on Semantic Correlation and Multi-Task Learning for Text-Video Retrieval

Adversarial Multi-Grained Embedding Network for Cross-Modal Text-Video Retrieval

Memory Enhanced Embedding Learning for Cross-Modal Video-Text Retrieval

Joint embeddings with multimodal cues for video-text retrieval

A Multi-interaction Model with Cross-Branch Feature Fusion for Video-Text Retrieval.

A Fusion Encoder with Multi-Task Guidance for Cross-Modal Text–Image Retrieval in Remote Sensing

Coarse-to-fine dual-level attention for video-text cross modal retrieval

Visual Spatio-temporal Relation-enhanced Network for Cross-modal Text-Video Retrieval

Spatial-temporal Graphs for Cross-modal Text2Video Retrieval

Video and Audio are Images: A Cross-Modal Mixer for Original Data on Video-Audio Retrieval

Cross-modal Embeddings for Video and Audio Retrieval

Multi-Granularity and Multi-modal Feature Interaction Approach for Text Video Retrieval

Tencent Text-Video Retrieval: Hierarchical Cross-Modal Interactions with Multi-Level Representations

Text-Video Retrieval via Variational Multi-Modal Hypergraph Networks

A multi-modal fusion approach for measuring web video relatedness

MGSGA: Multi-grained and Semantic-Guided Alignment for Text-Video Retrieval

Based on Spatial and Temporal Implicit Semantic Relational Inference for Cross-Modal Retrieval