Abstract:The core of cross-modal matching is to accurately measure the similarity between different modalities in a unified representation space. However, compared to textual descriptions of a certain perspective, the visual modality has more semantic variations. So, images are usually associated with multiple textual captions in databases. Although popular symmetric embedding methods have explored numerous modal interaction approaches, they often learn toward increasing the average expression probability of multiple semantic variations within image embeddings. Consequently, information entropy in embeddings is increased, resulting in redundancy and decreased accuracy. In this work, we propose a Dynamic Visual Semantic Sub-Embeddings framework (DVSE) to reduce the information entropy. Specifically, we obtain a set of heterogeneous visual sub-embeddings through dynamic orthogonal constraint loss. To encourage the generated candidate embeddings to capture various semantic variations, we construct a mixed distribution and employ a variance-aware weighting loss to assign different weights to the optimization process. In addition, we develop a Fast Re-ranking strategy (FR) to efficiently evaluate the retrieval results and enhance the performance. We compare the performance with existing set-based method using four image feature encoders and two text feature encoders on three benchmark datasets: MSCOCO, Flickr30K and CUB Captions. We also show the role of different components by ablation studies and perform a sensitivity analysis of the hyperparameters. The qualitative analysis of visualized bidirectional retrieval and attention maps further demonstrates the ability of our method to encode semantic variations.

Beyond visual semantics: Exploring the role of scene text in image understanding

Image-text Retrieval via Preserving Main Semantics of Vision

Multi-Modal Reasoning Graph for Scene-Text Based Fine-Grained Image Classification and Retrieval

SEMScene: Semantic-Consistency Enhanced Multi-Level Scene Graph Matching for Image-Text Retrieval

Scene-text aware cross-modal retrieval based on semantic matching (ChinaMM2024)

Visual and semantic guided scene text retrieval

Multi-view and region reasoning semantic enhancement for image-text retrieval

Integrating Scene Text and Visual Appearance for Fine-Grained Image Classification

Cross-Modal Image-Text Retrieval with Semantic Consistency

Don't only Feel Read: Using Scene text to understand advertisements

Visual Semantic Reasoning for Image-Text Matching

Direction-Oriented Visual-semantic Embedding Model for Remote Sensing Image-text Retrieval

Understanding, Categorizing and Predicting Semantic Image-Text Relations

Visual-Semantic Matching by Exploring High-Order Attention and Distraction

StacMR: Scene-Text Aware Cross-Modal Retrieval

Dynamic Visual Semantic Sub-Embeddings and Fast Re-Ranking

Tag-based Semantic Features for Scene Image Classification

Unambiguous Scene Text Segmentation with Referring Expression Comprehension

Cross-modal Semantic Enhanced Interaction for Image-Sentence Retrieval

Composing Object Relations and Attributes for Image-Text Matching

Text-based Person Search in Full Images via Semantic-Driven Proposal Generation