Abstract:As a bridge between language and vision domains, cross-modal retrieval between images and texts is a hot research topic in recent years. It remains challenging because the current image representations usually lack semantic concepts in the corresponding sentence captions. To address this issue, we introduce an intuitive and interpretable model to learn a common embedding space for alignments between images and text descriptions. Specifically, our model first incorporates the semantic relationship information into visual and textual features by performing region or word relationship reasoning. Then it utilizes the gate and memory mechanism to perform global semantic reasoning on these relationship-enhanced features, select the discriminative information and gradually grow representations for the whole scene. Through the alignment learning, the learned visual representations capture key objects and semantic concepts of a scene as in the corresponding text caption. Experiments on MS-COCO [1] and Flickr30K [2] datasets validate that our method surpasses many recent state-of-the-arts with a clear margin. In addition to the effectiveness, our methods are also very efficient at the inference stage. Thanks to the effective overall representation learning with visual semantic reasoning, our methods can already achieve very strong performance by only relying on the simple inner-product to obtain similarity scores between images and captions. Experiments validate the proposed methods are more than 30-75 times faster than many recent methods with code public available. Instead of following the recent trend of using complex local matching strategies [3], [4], [5], [6] to pursue good performance while sacrificing efficiency, we show that the simple global matching strategy can still be very effective, efficient and achieve even better performance based on our framework.

Learning a Semantic Space for Modeling Images, Tags and Feelings in Cross-Media Search.

A Benchmark Dataset and Learning High-Level Semantic Embeddings of Multimedia for Cross-Media Retrieval.

Learning A Semantic Space from User'S Relevance Feedback for Image Retrieval

Facebook5k: A Novel Evaluation Resource Dataset for Cross-Media Search

Multiple Kernel Visual-Auditory Representation Learning for Retrieval

Crossmedia retrieval by learning rich semantic embeddings of multimedia

Learning Semantic Correlations for Cross-Media Retrieval.

Enhanced Isomorphic Semantic Representation For Cross-Media Retrieval

Image Retrieval Based on Fuzzy Semantic Relevance Matrix

Learning a Semantic Space by Deep Network for Cross-media Retrieval.

Cross-Media Retrieval: Concepts, Advances And Challenges

A Real-World Web Cross-Media Dataset Containing Images, Texts and Videos

Cross-Modal Image-Text Retrieval with Semantic Consistency

Measuring the Semantic Relatedness Between Images Using Social Tags.

Sparse semantic metric learning for image retrieval

Image-Text Embedding Learning Via Visual and Textual Semantic Reasoning.

A Deep Semantic Alignment Network for the Cross-Modal Image-Text Retrieval in Remote Sensing

A Novel Approach Towards Large Scale Cross-Media Retrieval.

Cross-Domain Feature Learning in Multimedia

Measuring semantic relatedness between Flickr images: from a social tag based view.

A semantic subspace learning method to exploit relevance feedback log data for image retrieval