Abstract:Text-based person retrieval aims to find interest person images based on textual descriptions. The primary challenge in this task stems from the semantic gap resulting from the difference in feature granularity between text (which is characterized by coarse-grained features) and images (which are known for their fine-grained features). Previous works have utilized attention mechanisms to align modalities or to acquire a uniform representation, aiming to bridge the semantic gap between text and images. However, these methods suffer from two limitations: 1) Attention-based methods overlook subtle yet valuable information. 2) There exists a significant granularity gap between modalities, making the learning of a uniform representation time-consuming. To address these issues, we propose a Modal Complementarity framework based on Multimodal Large Language Model (MLLM-MC), which designed prompts according to task characteristics and utilized the multimodal abilities of Multimodal Large Language Model (MLLM) to produce elaborate textual descriptions for images. The textual descriptions generated by MLLM are used as a complement to the visual modality, thereby expanding the text-to-image retrieval task to encompass text-to-composite-image retrieval. To extract more comprehensive feature information, MLLM-MC employs a dual-stream model structure, which incorporates separate feature extractors for both visual and textual modalities. These extractors are further categorized into basic and detailed extractors, enabling the capture of information across different levels of granularity. Furthermore, in order to address the modal gap, we propose an uncertainty modeling technique within the visual branch, aiming to improve the model's matching patterns from one-to-one to one-to-many manner. The features from modal fusion are aligned using a transformer-based fusion module and low-order multimodal alignment. We conducted extensive experiments on three public datasets to evaluate the proposed MLLM-MC, achieving competitive Rank-1 accuracy of 68.58%, 62.66%, and 52.50% on CUHK-PEDES, ICFG-PEDES, and RSTPReid, respectively.

Modal Complementarity Based on Multimodal Large Language Model for Text-Based Person Retrieval

See Finer, See More: Implicit Modality Alignment for Text-based Person Retrieval

MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs

Multi-path Exploration and Feedback Adjustment for Text-to-Image Person Retrieval

Towards Unified Text-based Person Retrieval: A Large-scale Multi-Attribute and Language Search Benchmark

Look Before You Leap: Improving Text-based Person Retrieval by Learning A Consistent Cross-modal Common Manifold

DCEL: Deep Cross-modal Evidential Learning for Text-Based Person Retrieval

Multi-level similarity learning for image-text retrieval

Adaptive Uncertainty-Based Learning for Text-Based Person Retrieval

Cross-modal Co-occurrence Attributes Alignments for Person Search by Language

Hybrid Attention Network for Language-Based Person Search

Text-based person search via cross-modal alignment learning

Toward Robust Multimodal Learning using Multimodal Foundational Models

MoTaDual: Modality-Task Dual Alignment for Enhanced Zero-shot Composed Image Retrieval

Multi-granularity Separation Network for Text-Based Person Retrieval with Bidirectional Refinement Regularization.

BCRA: bidirectional cross-modal implicit relation reasoning and aligning for text-to-image person retrieval

Multi-Modal Retrieval For Large Language Model Based Speech Recognition

2M-NER: Contrastive Learning for Multilingual and Multimodal NER with Language and Modal Fusion

SCMM: Calibrating Cross-modal Representations for Text-Based Person Search

Cross-modal Uncertainty Modeling with Diffusion-based Refinement for Text-based Person Retrieval

Enhance the Robustness of Text-Centric Multimodal Alignments