Abstract:Learning common subspace is prevalent way in cross-modal retrieval to solve the problem of data from different modalities having inconsistent distributions and representations that cannot be directly compared. Previous cross-modal retrieval methods focus on projecting the cross-modal data into a common space by learning the correlation between them to bridge the modality gap. However, the rich semantic information in the video and the heterogeneous nature of audio-visual data leads to more serious heterogeneous gaps intuitively, which may lead to the loss of key semantic content of video with single clue by the previous methods when eliminating the modality gap, while the semantics of the categories may undermine the properties of the original features. In this work, we aim to learn effective audio-visual representations to support audio-visual cross-modal retrieval (AVCMR). We propose a novel model that maps audio-visual modalities into two distinct shared latent subspaces: explicit and implicit shared spaces. In particular, the explicit shared space is used to optimize pairwise correlations, where learned representations across modalities capture the commonalities of audio-visual pairs and reduce the modality gap. The implicit shared space is used to preserve the distinctive features between modalities by maintaining the discrimination of audio/video patterns from different semantic categories. Finally, the fusion of the features learned from the two latent subspaces is used for the similarity computation of the AVCMR task. The comprehensive experimental results on two audio-visual datasets demonstrate that our proposed model for using two different latent subspaces for audio-visual cross-modal learning is effective and significantly outperforms the state-of-the-art cross-modal models that learn features from a single subspace.

Cross-modal correlation learning for clustering on image-audio dataset.

Multiple Kernel Visual-Auditory Representation Learning for Retrieval

Understanding Visual-Auditory Correlation from Heterogeneous Features for Cross-Media Retrieval

Multimedia Data Clustering Based on Correlation Matrix Fusion

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Cross-Modal Subspace Clustering Via Deep Canonical Correlation Analysis.

Bridging the gap between visual and auditory feature spaces for cross-media retrieval

Cross-Modal Remote Sensing Image-Audio Retrieval with Adaptive Learning for Aligning Correlation

Multimedia Feature Mapping and Correlation Learning for Cross-Modal Retrieval

Cross-Modal Image Clustering Via Canonical Correlation Analysis

Cross-Modality Retrieval by Joint Correlation Learning

Learning Semantic Correlations for Cross-Media Retrieval.

Audio-Visual Embedding for Cross-Modal MusicVideo Retrieval through Supervised Deep CCA

A Cross-Media Distance Metric Learning Framework Based on Multi-View Correlation Mining and Matching

Multi-scale network with shared cross-attention for audio–visual correlation learning

Cluster-sensitive Structured Correlation Analysis for Web cross-modal retrieval

Adversarial Learning-Based Semantic Correlation Representation for Cross-Modal Retrieval

Cross-modal Deep Learning Applications: Audio-Visual Retrieval

Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis

Exploiting Correlation Consensus: Towards Subspace Clustering For Multi-Modal Data

Cross-Media Retrieval Method Based on Content Correlations