Abstract:Learning common subspace is prevalent way in cross-modal retrieval to solve the problem of data from different modalities having inconsistent distributions and representations that cannot be directly compared. Previous cross-modal retrieval methods focus on projecting the cross-modal data into a common space by learning the correlation between them to bridge the modality gap. However, the rich semantic information in the video and the heterogeneous nature of audio-visual data leads to more serious heterogeneous gaps intuitively, which may lead to the loss of key semantic content of video with single clue by the previous methods when eliminating the modality gap, while the semantics of the categories may undermine the properties of the original features. In this work, we aim to learn effective audio-visual representations to support audio-visual cross-modal retrieval (AVCMR). We propose a novel model that maps audio-visual modalities into two distinct shared latent subspaces: explicit and implicit shared spaces. In particular, the explicit shared space is used to optimize pairwise correlations, where learned representations across modalities capture the commonalities of audio-visual pairs and reduce the modality gap. The implicit shared space is used to preserve the distinctive features between modalities by maintaining the discrimination of audio/video patterns from different semantic categories. Finally, the fusion of the features learned from the two latent subspaces is used for the similarity computation of the AVCMR task. The comprehensive experimental results on two audio-visual datasets demonstrate that our proposed model for using two different latent subspaces for audio-visual cross-modal learning is effective and significantly outperforms the state-of-the-art cross-modal models that learn features from a single subspace.

Joint Audio-Visual Bi-Modal Codewords for Video Event Detection.

Discovering Joint Audio–visual Codewords for Video Event Detection

Specialty may be better: A decoupling multi-modal fusion network for Audio-visual event localization

Multimodal feature fusion for robust event detection in web videos

Modality Mixture Projections for Semantic Video Event Detection

Joint embeddings with multimodal cues for video-text retrieval

Integrating both Visual and Audio Cues for Enhanced Video Caption

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Multimodal Deep Representation Learning for Video Classification

Joint Visual and Audio Learning for Video Highlight Detection

Video and Audio are Images: A Cross-Modal Mixer for Original Data on Video-Audio Retrieval

Multimodality-guided Visual-Caption Semantic Enhancement

Event-centric multi-modal fusion method for dense video captioning

Audio-Visual Event Localization Via Recursive Fusion by Joint Co-Attention

Visual Word Proximity and Linguistics for Semantic Video Indexing and Near-Duplicate Retrieval

Integration of Multimodal Features for Video Scene Classification Based on HMM

Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing

Rethinking the constraints of multimodal fusion: case study in Weakly-Supervised Audio-Visual Video Parsing

CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation

Coordinated Joint Multimodal Embeddings for Generalized Audio-Visual Zeroshot Classification and Retrieval of Videos

Video Sentiment Analysis with Bimodal Information-augmented Multi-Head Attention