Abstract:Image-text retrieval is a central problem for understanding the semantic relationship between vision and language, and serves as the basis for various visual and language tasks. Most previous works either simply learn coarse-grained representations of the overall image and text, or elaborately establish the correspondence between image regions or pixels and text words. However, the close relations between coarse- and fine-grained representations for each modality are important for image-text retrieval but almost neglected. As a result, such previous works inevitably suffer from low retrieval accuracy or heavy computational cost. In this work, we address image-text retrieval from a novel perspective by combining coarse- and fine-grained representation learning into a unified framework. This framework is consistent with human cognition, as humans simultaneously pay attention to the entire sample and regional elements to understand the semantic content. To this end, a Token-Guided Dual Transformer (TGDT) architecture which consists of two homogeneous branches for image and text modalities, respectively, is proposed for image-text retrieval. The TGDT incorporates both coarse- and fine-grained retrievals into a unified framework and beneficially leverages the advantages of both retrieval approaches. A novel training objective called Consistent Multimodal Contrastive (CMC) loss is proposed accordingly to ensure the intra- and inter-modal semantic consistencies between images and texts in the common embedding space. Equipped with a two-stage inference method based on the mixed global and local cross-modal similarity, the proposed method achieves state-of-the-art retrieval performances with extremely low inference time when compared with representative recent approaches. Code is publicly available: github.com/LCFractal/TGDT.

Improving Cross-Modal Image-Text Retrieval With Teacher-Student Learning

Image-Text Retrieval Via Contrastive Learning with Auxiliary Generative Features and Support-set Regularization

Cross-modal Contrastive Learning for Generalizable and Efficient Image-text Retrieval

CODER: Coupled Diversity-Sensitive Momentum Contrastive Learning for Image-Text Retrieval

Cross-Modal Image-Text Retrieval with Semantic Consistency

Multi-level similarity learning for image-text retrieval

Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models

Cross-Graph Attention Enhanced Multi-Modal Correlation Learning for Fine-Grained Image-Text Retrieval

Image-Text Retrieval with Cross-Modal Semantic Importance Consistency.

Cross-modal Image-Text Retrieval with Multitask Learning

Efficient Token-Guided Image-Text Retrieval With Consistent Multimodal Contrastive Training

Hybrid Deep Neural Network-Based Cross-Modal Image and Text Retrieval Method for Large-Scale Data.

CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval

Iterative Uni-modal and Cross-modal Clustered Contrastive Learning for Image-text Retrieval

Cross-modal Image Retrieval with Deep Mutual Information Maximization

Cross-modality interaction reasoning for enhancing vision-language pre-training in image-text retrieval

EduCross: Dual adversarial bipartite hypergraph learning for cross-modal retrieval in multimodal educational slides

Towards Cross-Modal Text-Molecule Retrieval with Better Modality Alignment

Cross-Modal Learning With Images, Texts And Their Semantics

Visual context learning based on textual knowledge for image-text retrieval

Improving Multi-Modal Learning with Uni-Modal Teachers