Abstract:The core of cross-modal matching is to accurately measure the similarity between different modalities in a unified representation space. However, compared to textual descriptions of a certain perspective, the visual modality has more semantic variations. So, images are usually associated with multiple textual captions in databases. Although popular symmetric embedding methods have explored numerous modal interaction approaches, they often learn toward increasing the average expression probability of multiple semantic variations within image embeddings. Consequently, information entropy in embeddings is increased, resulting in redundancy and decreased accuracy. In this work, we propose a Dynamic Visual Semantic Sub-Embeddings framework (DVSE) to reduce the information entropy. Specifically, we obtain a set of heterogeneous visual sub-embeddings through dynamic orthogonal constraint loss. To encourage the generated candidate embeddings to capture various semantic variations, we construct a mixed distribution and employ a variance-aware weighting loss to assign different weights to the optimization process. In addition, we develop a Fast Re-ranking strategy (FR) to efficiently evaluate the retrieval results and enhance the performance. We compare the performance with existing set-based method using four image feature encoders and two text feature encoders on three benchmark datasets: MSCOCO, Flickr30K and CUB Captions. We also show the role of different components by ablation studies and perform a sensitivity analysis of the hyperparameters. The qualitative analysis of visualized bidirectional retrieval and attention maps further demonstrates the ability of our method to encode semantic variations.

Deep Multi-View Representation Learning for Social Images

Learning Disentangled Representation for Cross-Modal Retrieval with Deep Mutual Information Estimation.

Multimodal Learning of Social Image Representation by Exploiting Social Relations

Multi-View 3d Object Retrieval with Deep Embedding Network

Multi-view visual semantic embedding for cross-modal image–text retrieval

Semi-Supervised Multi-View Deep Discriminant Representation Learning.

Generalized Multi-view Embedding for Visual Recognition and Cross-modal Retrieval

Deep Attentive Multimodal Network Representation Learning for Social Media Images

Embedding Regularizer Learning for Multi-View Semi-Supervised Classification.

SkeletonNet: A Hybrid Network with a Skeleton-Embedding Process for Multi-View Image Representation Learning

Semantically consistent multi-view representation learning

Learning Social Image Embedding with Deep Multimodal Attention Networks

Deep Embedded Complementary and Interactive Information for Multi-View Classification

Multi-view Low-rank Preserving Embedding: A novel method for multi-view representation

Dynamic Visual Semantic Sub-Embeddings and Fast Re-Ranking

Multi-View Joint Graph Representation Learning for Urban Region Embedding

Embedded Deep Bilinear Interactive Information and Selective Fusion for Multi-view Learning

Multiview Concept Learning Via Deep Matrix Factorization

Learning Socially Embedded Visual Representation from Scratch

Low-Rank Discriminant Embedding for Multiview Learning.