Abstract:In this paper, we study the cross-modal image retrieval, where the inputs contain a source image plus some text that describes certain modifications to this image and the desired image. Prior work usually uses a three-stage strategy to tackle this task: 1) extracting the features of the inputs; 2) fusing the features of the source image and its modified text to obtain the fusion feature; 3) learning a similarity metric between the desired image and the source image plus modified text via deep metric learning. Since classical image/text encoders can learn useful representations and common pair-based loss functions of distance metric learning are enough for cross-modal retrieval, people usually improve retrieval accuracy by designing new fusion networks. However, these methods do not successfully handle the modality gap caused by the inconsistent feature distributions of different modalities, which greatly influences the feature fusion and the similarity learning. To alleviate this problem, we apply the contrastive self-supervised learning method Deep InfoMax (DIM) [1] to our approach to bridge this gap by enhancing the dependence between the text, the image, and their fusion. Specifically, our method narrows the modality gap between the text modality and the image modality by maximizing mutual information between their semantically inconsistent representations. Moreover, we seek an effective common subspace for the semantically consistent features of the fusion and the desired images by utilizing Deep InfoMax between the low-level layer of the image encoder and the high-level layer of the fusion network. Extensive experiments on three large-scale benchmarks show that we have bridged the modality gap between different modalities and achieve the state-of-the-art retrieval performance.

Cross Modal Retrieval Algorithm Based on Iterative Queries

Learning Disentangled Representation for Cross-Modal Retrieval with Deep Mutual Information Estimation.

A Cross-Modal Image and Text Retrieval Method Based on Efficient Feature Extraction and Interactive Learning CAE

Two-stage deep learning for supervised cross-modal retrieval

Effective Deep Learning-Based Multi-Modal Retrieval

Hybrid DAER Based Cross-modal Retrieval Exploiting Deep Representation Learning

The State of the Art for Cross-Modal Retrieval: A Survey

Hybrid Deep Neural Network-Based Cross-Modal Image and Text Retrieval Method for Large-Scale Data.

Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions

Deep Supervised Cross-Modal Retrieval

Cross-modal Image Retrieval with Deep Mutual Information Maximization

Semi-supervised Cross-Modal Learning for Cross Modal Retrieval and Image Annotation.

Is Cross-modal Information Retrieval Possible without Training?

Cross-Modal Retrieval by Class Information and Listwise Ranking

Improving Cross-Modal Image-Text Retrieval With Teacher-Student Learning

Cross‐modal retrieval with dual multi‐angle self‐attention

Real-world Cross-modal Retrieval Via Sequential Learning

Cross-Modal Retrieval Using Multiordered Discriminative Structured Subspace Learning.

Multi-task Framework Based on Feature Separation and Reconstruction for Cross-Modal Retrieval

Cross‐modal Semantic Correlation Learning by Bi‐CNN Network