Abstract:The core of cross-modal matching is to accurately measure the similarity between different modalities in a unified representation space. However, compared to textual descriptions of a certain perspective, the visual modality has more semantic variations. So, images are usually associated with multiple textual captions in databases. Although popular symmetric embedding methods have explored numerous modal interaction approaches, they often learn toward increasing the average expression probability of multiple semantic variations within image embeddings. Consequently, information entropy in embeddings is increased, resulting in redundancy and decreased accuracy. In this work, we propose a Dynamic Visual Semantic Sub-Embeddings framework (DVSE) to reduce the information entropy. Specifically, we obtain a set of heterogeneous visual sub-embeddings through dynamic orthogonal constraint loss. To encourage the generated candidate embeddings to capture various semantic variations, we construct a mixed distribution and employ a variance-aware weighting loss to assign different weights to the optimization process. In addition, we develop a Fast Re-ranking strategy (FR) to efficiently evaluate the retrieval results and enhance the performance. We compare the performance with existing set-based method using four image feature encoders and two text feature encoders on three benchmark datasets: MSCOCO, Flickr30K and CUB Captions. We also show the role of different components by ablation studies and perform a sensitivity analysis of the hyperparameters. The qualitative analysis of visualized bidirectional retrieval and attention maps further demonstrates the ability of our method to encode semantic variations.

Difficulty guided image retrieval using linear multiview embedding.

Difficulty Guided Image Retrieval Using Linear Multiple Feature Embedding

Image retrieval based on incremental subspace learning

Learning Disentangled Representation for Cross-Modal Retrieval with Deep Mutual Information Estimation.

Learning an Image Manifold for Retrieval

Multiview Locally Linear Embedding for Effective Medical Image Retrieval.

Query Difficulty Estimation for Image Retrieval

Random Projection Tree And Multiview Embedding For Large-Scale Image Retrieval

Multi-Query Parallel Field Ranking for image retrieval.

Query Difficulty Guided Image Retrieval System

Image retrieval using dimensionality reduction

Multiview Spectral Embedding

Discriminative Multi-View Interactive Image Re-Ranking.

Generalized Multi-view Embedding for Visual Recognition and Cross-modal Retrieval

Learning to Combine Ad-hoc Ranking Functions for Image Retrieval

Effective Image Retrieval Via Multilinear Multi-index Fusion.

ROMIR: Robust Multi-View Image Re-Ranking

From Document to Image: Learning a Scalable Ranking Model for Content Based Image Retrieval

Web image retrieval reranking with multi-view clustering.

Dynamic Visual Semantic Sub-Embeddings and Fast Re-Ranking

Low-Rank Discriminant Embedding for Multiview Learning.