Abstract:Learning common subspace is prevalent way in cross-modal retrieval to solve the problem of data from different modalities having inconsistent distributions and representations that cannot be directly compared. Previous cross-modal retrieval methods focus on projecting the cross-modal data into a common space by learning the correlation between them to bridge the modality gap. However, the rich semantic information in the video and the heterogeneous nature of audio-visual data leads to more serious heterogeneous gaps intuitively, which may lead to the loss of key semantic content of video with single clue by the previous methods when eliminating the modality gap, while the semantics of the categories may undermine the properties of the original features. In this work, we aim to learn effective audio-visual representations to support audio-visual cross-modal retrieval (AVCMR). We propose a novel model that maps audio-visual modalities into two distinct shared latent subspaces: explicit and implicit shared spaces. In particular, the explicit shared space is used to optimize pairwise correlations, where learned representations across modalities capture the commonalities of audio-visual pairs and reduce the modality gap. The implicit shared space is used to preserve the distinctive features between modalities by maintaining the discrimination of audio/video patterns from different semantic categories. Finally, the fusion of the features learned from the two latent subspaces is used for the similarity computation of the AVCMR task. The comprehensive experimental results on two audio-visual datasets demonstrate that our proposed model for using two different latent subspaces for audio-visual cross-modal learning is effective and significantly outperforms the state-of-the-art cross-modal models that learn features from a single subspace.

Multimodal visual dictionary learning via heterogeneous latent semantic sparse coding

Joint Dictionary Learning and Semantic Constrained Latent Subspace Projection for Cross-Modal Retrieval.

Multimodal Task-Driven Dictionary Learning for Image Classification

Laplacian Sparse Coding Dictionary for Image Set Based Collaborative Representation

Minimizing Dataset Bias: Discriminative Multi-Task Sparse Coding Through Shared Subspace Learning for Image Classification

Multi-level Discriminative Dictionary Learning with Application to Large Scale Image Classification.

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Heterogeneous Visual Features Fusion via Sparse Multimodal Machine

Cross-Modal Learning With Images, Texts And Their Semantics

Large Scale Visual Classification Via Learned Dictionaries And Sparse Representation

Learning a Multi-class Discriminative Dictionary with Nonredundancy Constraints for Visual Classification

Spatial-aware hyperspectral image classification via multifeature kernel dictionary learning

Semi-Supervised Coupled Dictionary Learning For Cross-Modal Retrieval In Internet Images And Texts

Multimodal sparse representation learning and applications

Discriminative Latent Feature Space Learning for Cross-Modal Retrieval

Latent Semantic Sparse Hashing for Cross-Modal Similarity Search.

Supervised Multi-Modal Dictionary Learning for Clothing Representation

Mutually Exclusive-Ksvd: Learning a Discriminative Dictionary for Hyperspectral Image Classification.

Supervised Dictionary Learning with Multiple Classifier Integration.

Class-Specific Sparse Principal Component Analysis for Visual Classification

Coupled dictionary learning and feature mapping for cross-modal retrieval