Abstract:Recently, generative adversarial network (GAN) has shown its strong ability on modeling data distribution via adversarial learning. Cross-modal GAN, which attempts to utilize the power of GAN to model the cross-modal joint distribution and to learn compatible cross-modal features, is becoming the research hotspot. However, the existing cross-modal GAN approaches typically 1) require labeled multimodal data of massive labor cost to establish cross-modal correlation; 2) utilize the vanilla GAN model that results in unstable training procedure and meaningless synthetic features; and 3) lack of extensibility for retrieving cross-modal data of new classes. In this article, we revisit the adversarial learning in existing cross-modal GAN methods and propose Joint Feature Synthesis and Embedding (JFSE), a novel method that jointly performs multimodal feature synthesis and common embedding space learning to overcome the above three shortcomings. Specifically, JFSE deploys two coupled conditional Wassertein GAN modules for the input data of two modalities, to synthesize meaningful and correlated multimodal features under the guidance of the word embeddings of class labels. Moreover, three advanced distribution alignment schemes with advanced cycle-consistency constraints are proposed to preserve the semantic compatibility and enable the knowledge transfer in the common embedding space for both the true and synthetic cross-modal features. All these add-ons in JFSE not only help to learn more effective common embedding space that effectively captures the cross-modal correlation but also facilitate to transfer knowledge to multimodal data of new classes. Extensive experiments are conducted on four widely used cross-modal datasets, and the comparisons with more than ten state-of-the-art approaches show that our JFSE method achieves remarkably accuracy improvement on both standard retrieval and the newly explored zero-shot and generalized zero-shot retrieval tasks.

Generative Adversarial and Self-Attention Based Fine-Grained Cross-Media Retrieval

Self-Attention based Fine-Grained Cross-Media Hybrid Network

CS-GAN: Centrosymmetric Generative Adversarial Network for Cross-Media Retrieval.

Multi-model Network for Fine-Grained Cross-Media Retrieval

Local Self-Attention on Fine-grained Cross-media Retrieval

Cross‐media search method based on complementary attention and generative adversarial network for social networks

A New Benchmark and Approach for Fine-grained Cross-media Retrieval

Deep Attentional Fine-Grained Similarity Network with Adversarial Learning for Cross-Modal Retrieval

ACE: A Generative Cross-Modal Retrieval Framework with Coarse-To-Fine Semantic Modeling

Learning a Semantic Space by Deep Network for Cross-media Retrieval.

Multi-Feature Graph Attention Network for Cross-Modal Video-Text Retrieval

Structures Aware Fine-grained Contrastive Adversarial Hashing for Cross-media Retrieval

Multi-modal Fine-grained Retrieval with Local and Global Cross-Attention.

Large-scale cross-media retrieval by Heterogeneous Feature Augmentation

Coarse-to-fine dual-level attention for video-text cross modal retrieval

Cross-Graph Attention Enhanced Multi-Modal Correlation Learning for Fine-Grained Image-Text Retrieval

Cross-media Retrieval Based on Semi-Supervised Regularization and Correlation Learning

A Jointly Guided Deep Network for Fine-Grained Cross-Modal Remote Sensing Text–Image Retrieval

Joint Feature Synthesis and Embedding: Adversarial Cross-Modal Retrieval Revisited

Adversarial Multi-Grained Embedding Network for Cross-Modal Text-Video Retrieval

Semi-supervised Cross-Modal Retrieval with Graph-Based Semantic Alignment Network