Abstract:We study the task of image retrieval with text feedback, where a reference image and modification text are composed to retrieve the desired target image. To accomplish this goal, existing methods always get the multimodal representations through different feature encoders and then adopt different strategies to model the correlation between the composed inputs and the target image. However, the multimodal query brings more challenges as it requires not only the synergistic understanding of the semantics from the heterogeneous multimodal inputs but also the ability to accurately build the underlying semantic correlation existing in each inputs-target triplet, i.e., reference image, modification text, and target image. In this paper, we tackle these issues with a novel Align and Retrieve (AlRet) framework. First, our proposed methods employ the contrastive loss in the feature encoders to learn meaningful multimodal representation while making the subsequent correlation modeling process in a more harmonious space. Then we propose to learn the accurate correlation between the composed inputs and target image in a novel composition-and-decomposition paradigm. Specifically, the composition network couples the reference image and modification text into a joint representation to learn the correlation between the joint representation and target image. The decomposition network conversely decouples the target image into visual and text subspaces to exploit the underlying correlation between the target image with each query element. The composition-and-decomposition paradigm forms a closed loop, which can be optimized simultaneously to promote each other in the performance. Massive comparison experiments on three real-world datasets confirm the effectiveness of the proposed method.

Enhancing Historical Image Retrieval with Compositional Cues

Exploring Geometric Information in CNN for Image Retrieval.

Composition Based Semantic Scene Retrieval For Ancient Murals

Hierarchical Composition Learning for Composed Query Image Retrieval.

Composition Analysis-Based Relevance Ranking for Ancient Mural

Composition-Aware Image Aesthetics Assessment

Comprehensive Linguistic-Visual Composition Network for Image Retrieval

Representation learning of image composition for aesthetic prediction

ICC++: Explainable Image Retrieval for Art Historical Corpora using Image Composition Canvas

Image aesthetics enhancement using composition-based saliency detection

Learning Visual Composition through Improved Semantic Guidance

Creative and High-Quality Image Composition Based on a New Criterion

Align and Retrieve: Composition and Decomposition Learning in Image Retrieval with Text Feedback

Composing Photos Like a Photographer

Target-Guided Composed Image Retrieval

Composed Image Retrieval Via Cross Relation Network with Hierarchical Aggregation Transformer.

Compositional Image Retrieval via Instruction-Aware Contrastive Learning

Clip-Based Composition-Aware Image Cropping

A Deep Learning Network Based End-to-end Image Composition.

Decompose Semantic Shifts for Composed Image Retrieval

CLIP-Based Composed Image Retrieval with Comprehensive Fusion and Data Augmentation.