Abstract:Cross-modal retrieval aims to retrieve the pertinent samples across different modalities, which is important in numerous multimodal applications. It is challenging to correlate the multimodal data due to a large heterogeneous gap between distinct modalities. In this paper, we propose a Multimodal Adversarial Network (MAN) method to project the multimodal data into a common space wherein the similarities between different modalities can be directly computed by the same distance measurement. The proposed MAN consists of multiple modality-specific generators, a discriminator and a multimodal discriminant analysis (MDA) loss. With the adversarial learning, the generators are pitted against the discriminator to eliminate the cross-modal discrepancy. Furthermore, a novel MDA loss is proposed to preserve as much discrimination as possible into all available dimensions of the generated common representations. However, there are some problems in directly optimizing the MDA trace criterion. To be specific, the discriminant function will overemphasize 1) the large distances between already separated classes, 2) and the dominant eigenvalues. These problems may cause poor discrimination of the common representations. To solve these problems, we propose a between-class strategy and an eigenvalue strategy to weaken the largest between-class differences and the dominant eigenvalues, respectively. To the best of our knowledge, the proposed MAN could be one of the first works to specifically design for the multimodal representation learning (more than two modalities) with adversarial learning. To verify the effectiveness of the proposed method, extensive experiments are carried out on four widely-used multimodal databases comparing with 16 state-of-the-art approaches.

Cross-Modal Manifold Learning for Cross-modal Retrieval

Learning Visually Aligned Semantic Graph for Cross-Modal Manifold Matching.

Learning an Image Manifold for Retrieval

Cross-Modal and Multimodal Data Analysis Based on Functional Mapping of Spectral Descriptors and Manifold Regularization

Image-based 3D Model Retrieval Using Manifold Learning

Cross-media Manifold Learning for Image Retrieval & Annotation

Cross-Modal Retrieval Using Multiordered Discriminative Structured Subspace Learning.

Bidirectional-isomorphic Manifold Learning at Image Semantic Understanding & Representation.

Adversarial Cross-Modal Retrieval via Learning and Transferring Single-Modal Similarities

Manifold Learning Based Cross-media Retrieval: A Solution to Media Object Complementary Nature

HCMSL: Hybrid Cross-modal Similarity Learning for Cross-modal Retrieval

Manifold learning through locally linear reconstruction based on Euclidean distance

Multimodal Adversarial Network for Cross-Modal Retrieval

Cross‐modal Semantic Correlation Learning by Bi‐CNN Network

Multi-Modal Web Image Retrieval Using Manifold-Ranking

Multi-View Similarity Learning Of Manifold Data

Metric Based On Multi-Order Spaces For Cross-Modal Retrieval

Image Retrieval Algorithms Based on Manifold Learning

Scalable Deep Multimodal Learning for Cross-Modal Retrieval

Geometric Multimodal Learning Based on Local Signal Expansion for Joint Diagonalization

Multi-Manifold Deep Discriminative Cross-Modal Hashing for Medical Image Retrieval