Abstract:With the development of multimedia technology, effective cross-modal retrieval methods are increasingly demanded. The key point of cross-modal retrieval is analyzing the correlation of heterogeneous modalities. There are mainly two types of correlation: content correlation and semantic correlation. Semantic correlation is constructed at a high level of abstraction which is more close to the human understanding than content correlation. In this paper, we investigate a semantic model to construct the semantic correlation for cross-modal retrieval. We assume that the semantic correlation of multimedia data from different modalities can be conditionally generated by semantic concepts in a probabilistic generation framework. The cross-modal semantic generation model (CMSGM) is proposed based on this assumption. We consider three cases of the cross-modal retrieval task. The first is the ideal case that all manifest concepts exist in training data for constructing the correlation, and we propose manifest CMSGM (M-CMSGM) which directly uses CMSGM on the manifest semantic concepts for retrieval. The second is the case that there are no manifest concepts in training data, and latent CMSGM (L-CMSGM) based on latent semantic concepts is proposed for this case, where the latent semantic concepts are learned by asymmetric spectral clustering. The last is the most general case that some of the manifest concepts exist, and we combine M-CMSGM and L-CMSGM to get combinative CMSGM (C-CMSGM) to solve this case. Experimental results on Wikipedia featured articles and MIR Flickr show that our methods have better performance compared with previous state-of-the-art methods. And C-CMSGM can maintain good performance in the case that manifest concepts are lacking, which confirms the robustness and practicality of C-CMSGM.

Image Retrieval Based on Multi-Concept Detector and Semantic Correlation.

Semantic Image Retrieval Based on Multiple-Instance Learning

Combining Convolutional Neural Network and Markov Random Field for Semantic Image Retrieval

Multi-spectral Remote Sensing Image Retrieval Based on Semantic Extraction

ImageSaker : A Semantic-based Image Retrieval System Refining with Concept Model

Image Retrieval Based on Fuzzy Semantic Relevance Matrix

Graph-based Multi-Space Semantic Correlation Propagation for Video Retrieval

Concept-based Multimedia Information Semantic Retrieval Model

Markov Random Field for Image Concept Detection

Research on Multi-concept Learning Based on Inter-concept Relation

Learning Semantic Concepts for Image Retrieval using the Max-Min Posterior Pseudo-Probabilities

Analyzing semantic correlation for cross-modal retrieval

MultiVCRank with Applications to Image Retrieval.

Multilateral Semantic Relations Modeling for Image Text Retrieval

Large-scale Semantic Web Image Retrieval Using Bimodal Deep Learning Techniques

Enhancing Semantic Image Retrieval with Limited Labeled Examples Via Deep Learning.

A Probabilistic Semantic Model for Image Annotation and Multi-Modal Image Retrieval

Bimodal Learning for Multi-concept Image Query

Image Retrieval Based on Various Semantic Feature Fusion

An Effective Video Retrieval Approach Based on Multi-modality Concept Correlation Graph

Multi-model Image Retrieval Method Based on Rough Set Inference and Colour Mutual Information