Abstract:Learning common subspace is prevalent way in cross-modal retrieval to solve the problem of data from different modalities having inconsistent distributions and representations that cannot be directly compared. Previous cross-modal retrieval methods focus on projecting the cross-modal data into a common space by learning the correlation between them to bridge the modality gap. However, the rich semantic information in the video and the heterogeneous nature of audio-visual data leads to more serious heterogeneous gaps intuitively, which may lead to the loss of key semantic content of video with single clue by the previous methods when eliminating the modality gap, while the semantics of the categories may undermine the properties of the original features. In this work, we aim to learn effective audio-visual representations to support audio-visual cross-modal retrieval (AVCMR). We propose a novel model that maps audio-visual modalities into two distinct shared latent subspaces: explicit and implicit shared spaces. In particular, the explicit shared space is used to optimize pairwise correlations, where learned representations across modalities capture the commonalities of audio-visual pairs and reduce the modality gap. The implicit shared space is used to preserve the distinctive features between modalities by maintaining the discrimination of audio/video patterns from different semantic categories. Finally, the fusion of the features learned from the two latent subspaces is used for the similarity computation of the AVCMR task. The comprehensive experimental results on two audio-visual datasets demonstrate that our proposed model for using two different latent subspaces for audio-visual cross-modal learning is effective and significantly outperforms the state-of-the-art cross-modal models that learn features from a single subspace.

TIR/VIS cross-modality modelling via correlative subspace learning

Learning a Locality Preserving Subspace for Visual Recognition.

Tir/Vis Correlation For Liveness Detection In Face Recognition

Modality-transfer Generative Adversarial Network and Dual-Level Unified Latent Representation for Visible Thermal Person Re-Identification

Near-infrared and visible light face recognition: a comprehensive survey

Joint Color-irrelevant Consistency Learning and Identity-aware Modality Adaptation for Visible-infrared Cross Modality Person Re-identification.

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

A Similarity Inference Metric for RGB-Infrared Cross-Modality Person Re-identification

Discovering attention-guided cross-modality correlation for visible–infrared person re-identification

Learning Domain and Pose Invariance for Thermal-to-Visible Face Recognition

Deep Cross-View Reconstruction GAN Based on Correlated Subspace for Multi-View Transformation

Cross-Spectral Attention for Unsupervised RGB-IR Face Verification and Person Re-identification

Vista-Morph: Unsupervised Image Registration of Visible-Thermal Facial Pairs

Correlation-Guided Discriminative Cross-Modality Features Network for Infrared and Visible Image Fusion

Cross-Domain Identification for Thermal-to-Visible Face Recognition

IR and visible light face recognition

Learning mappings for face synthesis from near infrared to visual light images

Enhancing the discriminative feature learning for visible-thermal cross-modality person re-identification

Translation, Association and Augmentation: Learning Cross-Modality Re-Identification From Single-Modality Annotation

Cross-Modality Person Re-identification with Memory-Based Contrastive Embedding

Infrared and Visible Cross-Modal Image Retrieval Through Shared Features