Abstract:As a bridge between language and vision domains, cross-modal retrieval between images and texts is a hot research topic in recent years. It remains challenging because the current image representations usually lack semantic concepts in the corresponding sentence captions. To address this issue, we introduce an intuitive and interpretable model to learn a common embedding space for alignments between images and text descriptions. Specifically, our model first incorporates the semantic relationship information into visual and textual features by performing region or word relationship reasoning. Then it utilizes the gate and memory mechanism to perform global semantic reasoning on these relationship-enhanced features, select the discriminative information and gradually grow representations for the whole scene. Through the alignment learning, the learned visual representations capture key objects and semantic concepts of a scene as in the corresponding text caption. Experiments on MS-COCO [1] and Flickr30K [2] datasets validate that our method surpasses many recent state-of-the-arts with a clear margin. In addition to the effectiveness, our methods are also very efficient at the inference stage. Thanks to the effective overall representation learning with visual semantic reasoning, our methods can already achieve very strong performance by only relying on the simple inner-product to obtain similarity scores between images and captions. Experiments validate the proposed methods are more than 30-75 times faster than many recent methods with code public available. Instead of following the recent trend of using complex local matching strategies [3], [4], [5], [6] to pursue good performance while sacrificing efficiency, we show that the simple global matching strategy can still be very effective, efficient and achieve even better performance based on our framework.

Learning Deep Structure-Preserving Image-Text Embeddings

Bilinear Joint Learning of Word and Entity Embeddings for Entity Linking.

Learning Two-Branch Neural Networks for Image-Text Matching Tasks.

Learning Structured Semantic Embeddings for Visual Recognition

Learning Robust Visual-Semantic Embeddings

Dual-path Convolutional Image-Text Embeddings with Instance Loss

Stacked Convolutional Deep Encoding Network for Video-Text Retrieval.

Image-Text Embedding Learning Via Visual and Textual Semantic Reasoning.

Learning Click-Based Deep Structure-Preserving Embeddings with Visual Attention.

Deep Multi-View Representation Learning for Social Images

Multimodal Deep Network Embedding with Integrated Structure and Attribute Information.

Part2Word: Learning Joint Embedding of Point Clouds and Text by Matching Parts to Words.

Expressing Objects just like Words: Recurrent Visual Embedding for Image-Text Matching

Hierarchical Multimodal Lstm For Dense Visual-Semantic Embedding

Deep Collaborative Embedding for Social Image Understanding.

Multi-Head Attention Driven Dynamic Visual-Semantic Embedding for Enhanced Image-Text Matching

Multi-level similarity learning for image-text retrieval

Multimodal Learning with Triplet Ranking Loss for Visual Semantic Embedding Learning

Learning Text-Image Joint Embedding for Efficient Cross-Modal Retrieval with Deep Feature Engineering

Learning Effective Word Embedding Using Morphological Word Similarity

Training and Evaluating Improved Dependency-Based Word Embeddings.