Abstract:Cross-modality recognition has many important applications in science, law enforcement and entertainment. Popular methods to bridge the modality gap include reducing the distributional differences of representations of different modalities, learning indistinguishable representations or explicit modality transfer. The first two approaches suffer from the loss of discriminant information while removing the modality-specific variations. The third one heavily relies on the successful modality transfer, could face catastrophic performance drop when explicit modality transfers are not possible or difficult. To tackle this problem, we proposed a compact encoder-decoder neural module (cmUNet) to learn modality-agnostic representations while retaining identity-related information. This is achieved through cross-modality transformation and in-modality reconstruction, enhanced by an adversarial/perceptual loss which encourages indistinguishability of representations in the original sample space. For cross-modality matching, we propose MarrNet where cmUNet is connected to a standard feature extraction network which takes as inputs the modality-agnostic representations and outputs similarity scores for matching. We validated our method on five challenging tasks, namely Raman-infrared spectrum matching, cross-modality person re-identification and heterogeneous (photo-sketch, visible-near infrared and visible-thermal) face recognition, where MarrNet showed superior performance compared to state-of-the-art methods. Furthermore, it is observed that a cross-modality matching method could be biased to extract discriminant information from partial or even wrong regions, due to incompetence of dealing with modality gaps, which subsequently leads to poor generalization. We show that robustness to occlusions can be an indicator of whether a method can well bridge the modality gap. This, to our knowledge, has been largely neglected in the previous works. Our experiments demonstrated that MarrNet exhibited excellent robustness against disguises and occlusions, and outperformed existing methods with a large margin (>10%). The proposed cmUNet is a meta-approach and can be used as a building block for various applications.

An Encoder Generative Adversarial Network for Multi-modality Image Recognition

X-Gacmn: An X-Shaped Generative Adversarial Cross-Modal Network With Hypersphere Embedding

Mind the Gap: Learning Modality-Agnostic Representations With a Cross-Modality UNet

A Multi-Scale Generative Adversarial Network for Real-World Image Denoising

Multi-Attention Generative Adversarial Network for Image Captioning.

Multi-scale Dual-Modal Generative Adversarial Networks for Text-to-image Synthesis

Autoencoder-Based Collaborative Attention GAN for Multi-Modal Image Synthesis

CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning

Feature Adversarial Network for Multimodal Template Matching

Multi-Pose Facial Expression Recognition Based on Generative Adversarial Network

Generative Adversarial Network with Multi-Branch Discriminator for Cross-Species Image-to-Image Translation

Cross-Modal Retrieval Based on Full-Modal Autoencoder with Generative Adversarial Mechanism

CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation

Multi-Task Adversarial Network for Disentangled Feature Learning

Modular Generative Adversarial Networks

Generative Adversarial Network with Multi-branch Discriminator for Imbalanced Cross-Species Image-to-image Translation.

Deep Memory Network for Cross-Modal Retrieval

Multi-expression Generative Adversarial Networks for Facial Expression Synthesis

Deep Supervised Dual Cycle Adversarial Network for Cross-Modal Retrieval

ACM: Adaptive Cross-Modal Graph Convolutional Neural Networks for RGB-D Scene Recognition.

Multimodal Unsupervised Image-to-Image Translation Without Independent Style Encoder