Abstract:Composed image retrieval aims to search a target image by concurrently understanding the composed inputs with a reference image and the complementary modification text. It aims to find a shared latent space where the representation of the composed inputs is close to the desired target image. Most previous methods capture the one-to-one correspondence between the composed inputs and target image, which encodes the composed inputs and the target image into single points in the feature space. However, the one-to-one correspondence cannot effectively handle this task due to the inherent ambiguity problem arising from the various semantic meanings and data uncertainty. Specifically, the composed inputs and target image always exhibit various semantic meanings, affecting the retrieval results. Moreover, given the composed inputs (resp. target image), there are multiple target images (resp. composed inputs) that equally make sense. In this paper, we propose a novel method termed Set of Diverse Queries with Uncertainty Regularization (SDQUR) to solve such inherent ambiguity problem. First, we utilize diverse queries to adaptively aggregate the composed inputs and target image into multiple deterministic embeddings that capture different semantic meanings in the triplet affecting the retrieval process. It can exploit the deterministic many-to-many correspondence within each triple through these set-based queries. Moreover, we provide an uncertainty regularization module to encode the composed inputs and target image into gaussian distribution. Multiple potential positive candidates are sampled from the distribution for probabilistic many-to-many correspondence. Through the complementary deterministic and probabilistic many-to-many correspondence manner, we achieve consistent improvements on the standard FashionIQ, CIRR, and Shoes benchmarks, surpassing the state-of-the-art methods by a large margin.

Answering Compositional Queries with Set-Theoretic Embeddings

Compositional Substitutivity of Visual Reasoning for Visual Question Answering

Query2box: Reasoning over Knowledge Graphs in Vector Space using Box Embeddings

Learning Probabilistic Box Embeddings for Effective and Efficient Ranking

A classification framework for multivariate compositional data with Dirichlet feature embedding

Word2Box: Capturing Set-Theoretic Semantics of Words using Box Embeddings

Thinking inside The Box: Learning Hypercube Representations for Group Recommendation

Neural-Symbolic Entangled Framework for Complex Query Answering

Viewpoint-Aware Representation for Sketch-Based 3D Model Retrieval

Concept2Box: Joint Geometric Embeddings for Learning Two-View Knowledge Graphs

Probabilistic Compositional Embeddings for Multimodal Image Retrieval

ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs

C-VQA: A Compositional Split of the Visual Question Answering (VQA) v1.0 Dataset

Multimodal Composition Example Mining for Composed Query Image Retrieval

Linear Spaces of Meanings: Compositional Structures in Vision-Language Models

Set of Diverse Queries with Uncertainty Regularization for Composed Image Retrieval

Composed Query Image Retrieval Using Locally Bounded Features

Hierarchical Composition Learning for Composed Query Image Retrieval.

Binder: Hierarchical Concept Representation through Order Embedding of Binary Vectors

Compositional Correlation Quantization for Large-Scale Multimodal Search.

Task2Box: Box Embeddings for Modeling Asymmetric Task Relationships