Abstract:Composed image retrieval (CIR) aims to retrieve the target image based on a multimodal query, i.e., a reference image paired with corresponding modification text. Recent CIR studies leverage vision-language pre-trained (VLP) methods as the feature extraction backbone, and perform nonlinear feature-level multimodal query fusion to retrieve the target image. Despite the promising performance, we argue that their nonlinear feature-level multimodal fusion may lead to the fused feature deviating from the original embedding space, potentially hurting the retrieval performance. To address this issue, in this work, we propose shifting the multimodal fusion from the feature level to the raw-data level to fully exploit the VLP model's multimodal encoding and cross-modal alignment abilities. In particular, we introduce a Dual Query Unification-based Composed Image Retrieval framework (DQU-CIR), whose backbone simply involves a VLP model's image encoder and a text encoder. Specifically, DQU-CIR first employs two training-free query unification components: text-oriented query unification and vision-oriented query unification, to derive a unified textual and visual query based on the raw data of the multimodal query, respectively. The unified textual query is derived by concatenating the modification text with the extracted reference image's textual description, while the unified visual query is created by writing the key modification words onto the reference image. Ultimately, to address diverse search intentions, DQU-CIR linearly combines the features of the two unified queries encoded by the VLP model to retrieve the target image. Extensive experiments on four real-world datasets validate the effectiveness of our proposed method.

What problem does this paper attempt to address?

The paper aims to address the issue of multimodal fusion in Composed Image Retrieval (CIR). Specifically, existing CIR methods typically use Vision-Language Pre-trained (VLP) models as the backbone for feature extraction and retrieve target images through nonlinear feature-level multimodal fusion. However, this approach may cause the fused features to deviate from the original embedding space, thereby affecting retrieval performance. To solve this problem, the paper proposes shifting multimodal fusion from the feature level to the raw data level to fully leverage the multimodal encoding and cross-modal alignment capabilities of VLP models. The paper designs a Dual Query Unification-based Composed Image Retrieval (DQU-CIR) framework, whose core idea is to convert multimodal queries into single-modal queries through two unsupervised query unification strategies—text-guided query unification and vision-guided query unification. The former concatenates the description of the reference image with the modification text into a unified text query; the latter directly writes the target image description words mentioned in the modification text onto the reference image to form a unified visual query. Both strategies are training-free and can be performed offline. Finally, DQU-CIR retrieves the target image by linearly weighting and combining the features of the two unified queries, ensuring that the fused multimodal query embeddings remain within the original embedding space of the VLP model, thereby improving retrieval performance.

Simple but Effective Raw-Data Level Multimodal Fusion for Composed Image Retrieval

Training-free Zero-shot Composed Image Retrieval via Weighted Modality Fusion and Similarity

Pseudo-triplet Guided Few-shot Composed Image Retrieval

Target-Guided Composed Image Retrieval

Compositional Image Retrieval via Instruction-Aware Contrastive Learning

Self-Training Boosted Multi-Factor Matching Network for Composed Image Retrieval

Geometry Sensitive Cross-Modal Reasoning for Composed Query Based Image Retrieval

MoTaDual: Modality-Task Dual Alignment for Enhanced Zero-shot Composed Image Retrieval

Image2Sentence based Asymmetrical Zero-shot Composed Image Retrieval

Multimodal Composition Example Mining for Composed Query Image Retrieval

Imagine and Seek: Improving Composed Image Retrieval with an Imagined Proxy

CaLa: Complementary Association Learning for Augmenting Composed Image Retrieval

Cross-Modal Attention Preservation with Self-Contrastive Learning for Composed Query-Based Image Retrieval

Fusion of Infrared and Visible Images Via Multi-Layer Convolutional Sparse Representation

VQA4CIR: Boosting Composed Image Retrieval with Visual Question Answering

Improving Composed Image Retrieval via Contrastive Learning with Scaling Positives and Negatives

Multi-Grained Attention Network with Mutual Exclusion for Composed Query-Based Image Retrieval

Spherical Linear Interpolation and Text-Anchoring for Zero-shot Composed Image Retrieval

Vision-by-Language for Training-Free Compositional Image Retrieval

DCFusion: Dual-Headed Fusion Strategy and Contextual Information Awareness for Infrared and Visible Remote Sensing Image

Composed Video Retrieval via Enriched Context and Discriminative Embeddings