Abstract:As a bridge between language and vision domains, cross-modal retrieval between images and texts is a hot research topic in recent years. It remains challenging because the current image representations usually lack semantic concepts in the corresponding sentence captions. To address this issue, we introduce an intuitive and interpretable model to learn a common embedding space for alignments between images and text descriptions. Specifically, our model first incorporates the semantic relationship information into visual and textual features by performing region or word relationship reasoning. Then it utilizes the gate and memory mechanism to perform global semantic reasoning on these relationship-enhanced features, select the discriminative information and gradually grow representations for the whole scene. Through the alignment learning, the learned visual representations capture key objects and semantic concepts of a scene as in the corresponding text caption. Experiments on MS-COCO [1] and Flickr30K [2] datasets validate that our method surpasses many recent state-of-the-arts with a clear margin. In addition to the effectiveness, our methods are also very efficient at the inference stage. Thanks to the effective overall representation learning with visual semantic reasoning, our methods can already achieve very strong performance by only relying on the simple inner-product to obtain similarity scores between images and captions. Experiments validate the proposed methods are more than 30-75 times faster than many recent methods with code public available. Instead of following the recent trend of using complex local matching strategies [3], [4], [5], [6] to pursue good performance while sacrificing efficiency, we show that the simple global matching strategy can still be very effective, efficient and achieve even better performance based on our framework.

Distributed Image Understanding with Semantic Dictionary and Semantic Expansion

Semantic Image Retrieval Based on Multiple-Instance Learning

Research on methodology of image semantic understanding based on generalized computing

Modeling Image Data for Effective Indexing and Retrieval in Large General Image Databases.

Cross-Modality Bridging and Knowledge Transferring for Image Understanding

Addressing Out-of-Distribution Challenges in Image Semantic Communication Systems with Multi-modal Large Language Models

Semantic context learning with large-scale weakly-labeled image set.

Fast Semantic Diffusion for Large-Scale Context-Based Image and Video Annotation

An Image-Semantic Ontological Framework For Large Image Databases

Deep discriminative image feature learning for cross-modal semantics understanding

Facilitating Image Search with a Scalable and Compact Semantic Mapping

Bidirectional-isomorphic Manifold Learning at Image Semantic Understanding & Representation.

Towards Multi-Semantic Image Annotation with Graph Regularized Exclusive Group Lasso

SSSIC: Semantics-to-Signal Scalable Image Coding with Learned Structural Representations.

Learning Semantic Embedding At A Large Scale

Integration of Semantic and Visual Hashing for Image Retrieval

Semantic Probability Distribution Modeling for Diverse Semantic Image Synthesis

Deep Semantic Understanding of High Resolution Remote Sensing Image

Exploiting textual queries for dynamically visual disambiguation

Image-Text Embedding Learning Via Visual and Textual Semantic Reasoning.

A Probabilistic Semantic Model for Image Annotation and Multi-Modal Image Retrieval