Abstract:Composed image retrieval, a task involving the search for a target image using a reference image and a complementary text as the query, has witnessed significant advancements owing to the progress made in cross-modal modeling. Unlike the general image-text retrieval problem with only one alignment relation, i.e., image-text, we argue for the existence of two types of relations in composed image retrieval. The explicit relation pertains to the reference image complementary text-target image, which is commonly exploited by existing methods. Besides this intuitive relation, the observations during our practice have uncovered another implicit yet crucial relation, i.e., reference image target image-complementary text, since we found that the complementary text can be inferred by studying the relation between the target image and the reference image. Regrettably, existing methods largely focus on leveraging the explicit relation to learn their networks, while overlooking the implicit relation. In response to this weakness, We propose a new framework for composed image retrieval, termed dual relation alignment, which integrates both explicit and implicit relations to fully exploit the correlations among the triplets. Specifically, we design a vision compositor to fuse reference image and target image at first, then the resulted representation will serve two roles: (1) counterpart for semantic alignment with the complementary text and (2) compensation for the complementary text to boost the explicit relation modeling, thereby implant the implicit relation into the alignment learning. Our method is evaluated on two popular datasets, CIRR and FashionIQ, through extensive experiments. The results confirm the effectiveness of our dual-relation learning in substantially enhancing composed image retrieval performance.

Deep compositional cross-modal learning to rank via local-global alignment

Deep Compositional Cross-modal Learning to Rank via Local-Global Alignment

Cross-Modal Learning to Rank Via Latent Joint Representation

Cross-modal Deep Metric Learning with Multi-Task Regularization

Ranking with local regression and global alignment for cross media retrieval.

A Low Rank Structural Large Margin Method for Cross-Modal Ranking

Learning Multimodal Neural Network with Ranking Examples

Ranking-Based Deep Cross-Modal Hashing

Semantic Pre-alignment and Ranking Learning with Unified Framework for Cross-modal Retrieval

Combinative Hypergraph Learning in Subspace for Cross-Modal Ranking.

Deep Cross-modal Hashing Based on Semantic Consistent Ranking

Learning Cross-Aligned Latent Embeddings for Zero-Shot Cross-Modal Retrieval

Cross-media semantic representation via bi-directional learning to rank.

Multi-Networks Joint Learning for Large-Scale Cross-Modal Retrieval

Special Issue on Cross-Modal Retrieval and Analysis

Simple to Complex Cross-modal Learning to Rank

Multi-level similarity learning for image-text retrieval

Dual Relation Alignment for Composed Image Retrieval

Deep Multi-Graph Hierarchical Enhanced Semantic Representation for Cross-Modal Retrieval

Cross-modal alignment with graph reasoning for image-text retrieval

Semantic enhancement and multi-level alignment network for cross-modal retrieval