Abstract:Cross-modal retrieval, e.g., using an image query to search related text and vice-versa, has become a highlighted research topic, to provide flexible retrieval experience across multi-modal data. Existing approaches usually consider the so-called non-extendable cross-modal retrieval task. In this task, they learn a common latent subspace from a source set containing labeled instances of image-text pairs and then generate common representation for the instances in a target set to perform cross-modal matching. However, these method may not generalize well when the instances of the target set contains unseen classes since the instances of both the source and target set are assumed to share the same range of classes in the non-extensive cross-modal retrieval task. In this paper, we consider a more practical issue of extendable cross-modal retrieval task where instances in source and target set have disjoint classes. We propose a novel framework, termed Modal-adversarial Semantic Learning Network (MASLN), to tackle the limitation of existing methods on this practical task. Specifically, the proposed MASLN consists two subnetworks of cross-modal reconstruction and modal-adversarial semantic learning. The former minimizes the cross-modal distribution discrepancy by reconstructing each modality data mutually, with the guidelines of class embeddings as side information in the reconstruction procedure. The latter generates semantic representation to be indiscriminative for modalities, while to distinguish the modalities from the common representation via an adversarial learning mechanism. The two subnetworks are jointly trained to enhance the cross-modal semantic consistency in the learned common subspace and the knowledge transfer to instances in the target set. Comprehensive experiment on three widely-used multi-modal datasets show its effectiveness and robustness on both non-extendable and extendable cross-modal retrieval task.

Learning Joint Multimodal Representation with Adversarial Attention Networks

Robust Multimodal Representation Learning With Evolutionary Adversarial Attention Networks

Multimodal Adversarially Learned Inference with Factorized Discriminators

Adversarial Multimodal Network for Movie Question Answering

Multimodal Network Embedding Via Attention Based Multi-view Variational Autoencoder.

Attention-Based Modality-Gated Networks for Image-Text Sentiment Analysis

Learning Joint Multimodal Representation Based On Multi-Fusion Deep Neural Networks

Multimodal Learning of Social Image Representation by Exploiting Social Relations

Adversarial Attentive Multi-Modal Embedding Learning for Image-Text Matching

Multimodal Representation Learning by Alternating Unimodal Adaptation

Modal-adversarial Semantic Learning Network for Extendable Cross-modal Retrieval

Object-Aware Multimodal Named Entity Recognition in Social Media Posts With Adversarial Learning

Adversarial-Metric Learning for Audio-Visual Cross-Modal Matching

Learning to Learn Better Unimodal Representations Via Adaptive Multimodal Meta-Learning

A 2 CMHNE: Attention-Aware Collaborative Multimodal Heterogeneous Network Embedding

Learning Social Image Embedding with Deep Multimodal Attention Networks

Adversarial Learning with Multi-Modal Attention for Visual Question Answering.

Multimodal Attentive Representation Learning for Micro-video Multi-label Classification

Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion

Attention Alignment Multimodal LSTM for Fine-Gained Common Space Learning.

Multi-Modal Representation via Contrastive Learning with Attention Bottleneck Fusion and Attentive Statistics Features