Abstract:Cross-modality recognition has many important applications in science, law enforcement and entertainment. Popular methods to bridge the modality gap include reducing the distributional differences of representations of different modalities, learning indistinguishable representations or explicit modality transfer. The first two approaches suffer from the loss of discriminant information while removing the modality-specific variations. The third one heavily relies on the successful modality transfer, could face catastrophic performance drop when explicit modality transfers are not possible or difficult. To tackle this problem, we proposed a compact encoder-decoder neural module (cmUNet) to learn modality-agnostic representations while retaining identity-related information. This is achieved through cross-modality transformation and in-modality reconstruction, enhanced by an adversarial/perceptual loss which encourages indistinguishability of representations in the original sample space. For cross-modality matching, we propose MarrNet where cmUNet is connected to a standard feature extraction network which takes as inputs the modality-agnostic representations and outputs similarity scores for matching. We validated our method on five challenging tasks, namely Raman-infrared spectrum matching, cross-modality person re-identification and heterogeneous (photo-sketch, visible-near infrared and visible-thermal) face recognition, where MarrNet showed superior performance compared to state-of-the-art methods. Furthermore, it is observed that a cross-modality matching method could be biased to extract discriminant information from partial or even wrong regions, due to incompetence of dealing with modality gaps, which subsequently leads to poor generalization. We show that robustness to occlusions can be an indicator of whether a method can well bridge the modality gap. This, to our knowledge, has been largely neglected in the previous works. Our experiments demonstrated that MarrNet exhibited excellent robustness against disguises and occlusions, and outperformed existing methods with a large margin (>10%). The proposed cmUNet is a meta-approach and can be used as a building block for various applications.

Modal-nexus auto-encoder for multi-modality cellular data integration and imputation

scMultiNODE: Integrative Model for Multi-Modal Temporal Single-Cell Data

A coupled autoencoder approach for multi-modal analysis of cell types

Joint variational autoencoders for multimodal imputation and embedding

Integrating single-cell multimodal epigenomic data using 1D-convolutional neural networks

Multimodal Single Cell Data Integration Challenge: Results and Lessons Learned

MIDAS: a deep generative model for mosaic integration and knowledge transfer of single-cell multimodal data

Joint Analysis of Single-Cell Data across Cohorts with Missing Modalities

Integrated analysis of multimodal single-cell data with structural similarity

Learn to Combine Modalities in Multimodal Deep Learning

CrossMP: Enabling Cross-Modality Translation between Single-Cell RNA-Seq and Single-Cell ATAC-Seq through Web-Based Portal

Partially Shared Multi-Modal Embedding Learns Holistic Representation of Cell State

Comprehensive View Embedding Learning for Single-Cell Multimodal Integration

scMODAL: A general deep learning framework for comprehensive single-cell multi-omics data alignment with feature links

Leveraging hierarchy in multimodal generative models for effective cross-modality inference

Graph Neural Networks for Multimodal Single-Cell Data Integration

Deep Fusion Of Heterogeneous Sensor Data

scCross: a deep generative model for unifying single-cell multi-omics with seamless integration, cross-modal generation, and in silico exploration

Advancing Multi-Modal Sensing Through Expandable Modality Alignment

Single-cell mosaic integration and cell state transfer with auto-scaling self-attention mechanism

Mind the Gap: Learning Modality-Agnostic Representations With a Cross-Modality UNet