Abstract:Cross-modal retrieval tasks, which are more natural and challenging than traditional retrieval tasks, have attracted increasing interest from researchers in recent years. Although different modalities with the same semantics have some potential relevance, the feature space heterogeneity still seriously weakens the performance of cross-modal retrieval models. To solve this problem, common space-based methods in which multimodal data is projected into a learned common space for similarity measurement have become the mainstream approach for cross-modal retrieval tasks. However, current methods entangle the modality style and semantic content in the common space and neglect to fully explore the semantic and discriminative representation/reconstruction of the semantic content. This often results in an unsatisfactory retrieval performance. To solve these issues, this paper proposes a new Deep Supervised Dual Cycle Adversarial Network (DSDCAN) model based on common space learning. It is composed of two cross-modal cycle GANs, one for the image and one for the text. The proposed cycle GAN model disentangles the semantic content and modality style features by making the data of one modality well reconstructed from the extracted modal style feature and the content feature of the other modality. Then, a discriminative semantic and label loss is proposed by fully considering the category, sample contrast, and label supervision to enhance the semantic discrimination of the common space representation. Besides this, to make the data distribution between two modalities similar, a second-order similarity is presented as a distance measurement of the cross-modal representation in the common space. Extensive experiments have been conducted on the Wikipedia, Pascal Sentence, NUS-WIDE-10k, PKU XMedia, MSCOCO, NUS-WIDE, Flickr30k and MIRFlickr datasets. The results demonstrate that the proposed method can achieve a higher performance than the state-of-the-art methods.

Cross-modal Discriminant Adversarial Network

X-Gacmn: An X-Shaped Generative Adversarial Cross-Modal Network With Hypersphere Embedding

Multimodal Adversarially Learned Inference with Factorized Discriminators

Dual discriminant adversarial cross-modal retrieval

Cross-Modal Search for Social Networks via Adversarial Learning

Deep Supervised Dual Cycle Adversarial Network for Cross-Modal Retrieval

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Cross-domain attention network for hyperspectral image classification

Adversarial Cross-Modal Retrieval

Deep Attentional Fine-Grained Similarity Network with Adversarial Learning for Cross-Modal Retrieval

Contrastive Adaptation Network for Single- and Multi-Source Domain Adaptation

Multicenter clinical trial of implanted norethindrone pellets for long-acting contraception in women. Program for Applied Research on Fertility Regulation.

CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning

Adversarial-Metric Learning for Audio-Visual Cross-Modal Matching

A Dimensional Structure based Knowledge Distillation Method for Cross-Modal Learning

A Generalization Theory of Cross-Modality Distillation with Contrastive Learning

Adversarial Cross-Modal Retrieval via Learning and Transferring Single-Modal Similarities

Deep cross-modal subspace clustering with Contrastive Neighbour Embedding

MCAN: Multimodal Causal Adversarial Networks for Dynamic Effective Connectivity Learning from fMRI and EEG Data

Discrimination-Aware Domain Adversarial Neural Network