Abstract:Image-text retrieval is a central problem for understanding the semantic relationship between vision and language, and serves as the basis for various visual and language tasks. Most previous works either simply learn coarse-grained representations of the overall image and text, or elaborately establish the correspondence between image regions or pixels and text words. However, the close relations between coarse- and fine-grained representations for each modality are important for image-text retrieval but almost neglected. As a result, such previous works inevitably suffer from low retrieval accuracy or heavy computational cost. In this work, we address image-text retrieval from a novel perspective by combining coarse- and fine-grained representation learning into a unified framework. This framework is consistent with human cognition, as humans simultaneously pay attention to the entire sample and regional elements to understand the semantic content. To this end, a Token-Guided Dual Transformer (TGDT) architecture which consists of two homogeneous branches for image and text modalities, respectively, is proposed for image-text retrieval. The TGDT incorporates both coarse- and fine-grained retrievals into a unified framework and beneficially leverages the advantages of both retrieval approaches. A novel training objective called Consistent Multimodal Contrastive (CMC) loss is proposed accordingly to ensure the intra- and inter-modal semantic consistencies between images and texts in the common embedding space. Equipped with a two-stage inference method based on the mixed global and local cross-modal similarity, the proposed method achieves state-of-the-art retrieval performances with extremely low inference time when compared with representative recent approaches. Code is publicly available: github.com/LCFractal/TGDT.

Coarse-to-fine Alignment Makes Better Speech-image Retrieval

Towards Fast and Accurate Image-Text Retrieval with Self-Supervised Fine-Grained Alignment

Improving Contrastive Learning of Sentence Embeddings with Focal-InfoNCE

A Reconstruction-based Visual-Acoustic-Semantic Embedding Method for Speech-Image Retrieval

ContextBLIP: Doubly Contextual Alignment for Contrastive Image Retrieval from Linguistically Complex Descriptions

Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learning

Image-Text Retrieval with Cross-Modal Semantic Importance Consistency.

Fuse after Align: Improving Face-Voice Association Learning via Multimodal Encoder

Contrastive Video-Language Learning with Fine-grained Frame Sampling

Cross-Modal Denoising: A Novel Training Paradigm for Enhancing Speech-Image Retrieval

Optimizing CLIP Models for Image Retrieval with Maintained Joint-Embedding Alignment

BrewCLIP: A Bifurcated Representation Learning Framework for Audio-Visual Retrieval

ITContrast: contrastive learning with hard negative synthesis for image-text matching

Advancing Multi-grained Alignment for Contrastive Language-Audio Pre-training

Unified Coarse-to-Fine Alignment for Video-Text Retrieval

Efficient Token-Guided Image-Text Retrieval With Consistent Multimodal Contrastive Training

Mutual Alignment between Audiovisual Features for End-to-End Audiovisual Speech Recognition

SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger.

Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models

Enhancing Sound Source Localization via False Negative Elimination

Coarse-to-Fine: Learning Compact Discriminative Representation for Single-Stage Image Retrieval