Abstract:Text-based person retrieval aims to find interest person images based on textual descriptions. The primary challenge in this task stems from the semantic gap resulting from the difference in feature granularity between text (which is characterized by coarse-grained features) and images (which are known for their fine-grained features). Previous works have utilized attention mechanisms to align modalities or to acquire a uniform representation, aiming to bridge the semantic gap between text and images. However, these methods suffer from two limitations: 1) Attention-based methods overlook subtle yet valuable information. 2) There exists a significant granularity gap between modalities, making the learning of a uniform representation time-consuming. To address these issues, we propose a Modal Complementarity framework based on Multimodal Large Language Model (MLLM-MC), which designed prompts according to task characteristics and utilized the multimodal abilities of Multimodal Large Language Model (MLLM) to produce elaborate textual descriptions for images. The textual descriptions generated by MLLM are used as a complement to the visual modality, thereby expanding the text-to-image retrieval task to encompass text-to-composite-image retrieval. To extract more comprehensive feature information, MLLM-MC employs a dual-stream model structure, which incorporates separate feature extractors for both visual and textual modalities. These extractors are further categorized into basic and detailed extractors, enabling the capture of information across different levels of granularity. Furthermore, in order to address the modal gap, we propose an uncertainty modeling technique within the visual branch, aiming to improve the model's matching patterns from one-to-one to one-to-many manner. The features from modal fusion are aligned using a transformer-based fusion module and low-order multimodal alignment. We conducted extensive experiments on three public datasets to evaluate the proposed MLLM-MC, achieving competitive Rank-1 accuracy of 68.58%, 62.66%, and 52.50% on CUHK-PEDES, ICFG-PEDES, and RSTPReid, respectively.

BCRA: bidirectional cross-modal implicit relation reasoning and aligning for text-to-image person retrieval

Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval

See Finer, See More: Implicit Modality Alignment for Text-based Person Retrieval

Masked Contrastive Reconstruction for Cross-modal Medical Image-Report Retrieval

A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues

Modal Complementarity Based on Multimodal Large Language Model for Text-Based Person Retrieval

Bi-Modal Progressive Mask Attention for Fine-Grained Recognition.

Masking-Based Cross-Modal Remote Sensing Image–Text Retrieval via Dynamic Contrastive Learning

Exploring Uni-Modal Feature Learning on Entities and Relations for Remote Sensing Cross-Modal Text-Image Retrieval

Attribute-Aware Implicit Modality Alignment for Text Attribute Person Search

Cross-Modal Adaptive Dual Association for Text-to-Image Person Retrieval

Rethinking referring relationships from a perspective of mask-level relational reasoning

Cascaded Cross-modal Alignment for Visible-Infrared Person Re-Identification

Multi-granularity Separation Network for Text-Based Person Retrieval with Bidirectional Refinement Regularization.

Multi-Modal Mutual Attention and Iterative Interaction for Referring Image Segmentation

IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrieval

DCEL: Deep Cross-modal Evidential Learning for Text-Based Person Retrieval

Masked Vision and Language Modeling for Multi-modal Representation Learning

Knowing Where to Focus: Attention-Guided Alignment for Text-based Person Search

RA-BLIP: Multimodal Adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training

Bidirectional Focused Semantic Alignment Attention Network for Cross-Modal Retrieval