Abstract:Cross-modality recognition has many important applications in science, law enforcement and entertainment. Popular methods to bridge the modality gap include reducing the distributional differences of representations of different modalities, learning indistinguishable representations or explicit modality transfer. The first two approaches suffer from the loss of discriminant information while removing the modality-specific variations. The third one heavily relies on the successful modality transfer, could face catastrophic performance drop when explicit modality transfers are not possible or difficult. To tackle this problem, we proposed a compact encoder-decoder neural module (cmUNet) to learn modality-agnostic representations while retaining identity-related information. This is achieved through cross-modality transformation and in-modality reconstruction, enhanced by an adversarial/perceptual loss which encourages indistinguishability of representations in the original sample space. For cross-modality matching, we propose MarrNet where cmUNet is connected to a standard feature extraction network which takes as inputs the modality-agnostic representations and outputs similarity scores for matching. We validated our method on five challenging tasks, namely Raman-infrared spectrum matching, cross-modality person re-identification and heterogeneous (photo-sketch, visible-near infrared and visible-thermal) face recognition, where MarrNet showed superior performance compared to state-of-the-art methods. Furthermore, it is observed that a cross-modality matching method could be biased to extract discriminant information from partial or even wrong regions, due to incompetence of dealing with modality gaps, which subsequently leads to poor generalization. We show that robustness to occlusions can be an indicator of whether a method can well bridge the modality gap. This, to our knowledge, has been largely neglected in the previous works. Our experiments demonstrated that MarrNet exhibited excellent robustness against disguises and occlusions, and outperformed existing methods with a large margin (>10%). The proposed cmUNet is a meta-approach and can be used as a building block for various applications.

X-Gacmn: An X-Shaped Generative Adversarial Cross-Modal Network With Hypersphere Embedding

CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning

CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

MHTN: Modal-Adversarial Hybrid Transfer Network for Cross-Modal Retrieval

Weighted Graph-structured Semantics Constraint Network for Cross-Modal Retrieval

Multimodal Adversarially Learned Inference with Factorized Discriminators

Adversarial Cross-Modal Retrieval

Deep Supervised Dual Cycle Adversarial Network for Cross-Modal Retrieval

Adversarial Graph Convolutional Network for Cross-Modal Retrieval

Integrating Multi-Label Contrastive Learning With Dual Adversarial Graph Neural Networks for Cross-Modal Retrieval

Cross-Modal Generation and Pair Correlation Alignment Hashing

Achieving Cross Modal Generalization with Multimodal Unified Representation.

Zero-Shot Cross-Media Embedding Learning with Dual Adversarial Distribution Network.

Towards cross-modal organ translation and segmentation: A cycle- and shape-consistent generative adversarial network

SCH-GAN: Semi-Supervised Cross-Modal Hashing by Generative Adversarial Network

Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion

Partial Modal Conditioned GANs for Multi-modal Multi-label Learning with Arbitrary Modal-Missing

Dual Adversarial Graph Neural Networks for Multi-label Cross-modal Retrieval

Mind the Gap: Learning Modality-Agnostic Representations With a Cross-Modality UNet

Cross-Domain Hyperspectral Image Classification Based on Generative Adversarial Networks