Abstract:As a bridge between language and vision domains, cross-modal retrieval between images and texts is a hot research topic in recent years. It remains challenging because the current image representations usually lack semantic concepts in the corresponding sentence captions. To address this issue, we introduce an intuitive and interpretable model to learn a common embedding space for alignments between images and text descriptions. Specifically, our model first incorporates the semantic relationship information into visual and textual features by performing region or word relationship reasoning. Then it utilizes the gate and memory mechanism to perform global semantic reasoning on these relationship-enhanced features, select the discriminative information and gradually grow representations for the whole scene. Through the alignment learning, the learned visual representations capture key objects and semantic concepts of a scene as in the corresponding text caption. Experiments on MS-COCO [1] and Flickr30K [2] datasets validate that our method surpasses many recent state-of-the-arts with a clear margin. In addition to the effectiveness, our methods are also very efficient at the inference stage. Thanks to the effective overall representation learning with visual semantic reasoning, our methods can already achieve very strong performance by only relying on the simple inner-product to obtain similarity scores between images and captions. Experiments validate the proposed methods are more than 30-75 times faster than many recent methods with code public available. Instead of following the recent trend of using complex local matching strategies [3], [4], [5], [6] to pursue good performance while sacrificing efficiency, we show that the simple global matching strategy can still be very effective, efficient and achieve even better performance based on our framework.

Leveraging Conceptualization for Short-Text Embedding

Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings

Incorporating Knowledge into Neural Network for Text Representation.

Conceptual Sentence Embeddings

Improving Chinese Word Representation with Conceptual Semantics

Concept-based Short Text Classification and Ranking

Learning New Word Semantics with Conceptual Text.

A Hybrid Semantic Representation Method Based on Fusion Conceptual Knowledge and Weighted Word Embeddings for English Texts

Short Text Understanding Based on Conceptual and Semantic Enrichment.

Concept Representation by Learning Explicit and Implicit Concept Couplings

Semantic Structure-Based Word Embedding by Incorporating Concept Convergence and Word Divergence

Distilling Semantic Concept Embeddings from Contrastively Fine-Tuned Language Models

Learning Concept Embeddings for Efficient Bag-of-Concepts Densification

Enhancing Semantic Word Representations by Embedding Deeper Word Relationships

Topic Discovery for Short Texts Using Word Embeddings

Collaboratively Modeling and Embedding of Latent Topics for Short Texts

Leveraging Pattern Associations For Word Embedding Models

Extending Embedding Representation by Incorporating Latent Relations.

Image-Text Embedding Learning Via Visual and Textual Semantic Reasoning.

Concept Embedded Convolutional Semantic Model for Question Retrieval.

A Semantic Modeling Method for Social Network Short Text Based on Spatial and Temporal Characteristics