Abstract:How do we retrieve images accurately? Also, how do we rank a group of images precisely and efficiently for specific queries? These problems are critical for researchers and engineers to generate a novel image searching engine. First, it is important to obtain an appropriate description that effectively represent the images. In this paper, multimodal features are considered for describing images. The images unique properties are reflected by visual features, which are correlated to each other. However, semantic gaps always exist between images visual features and semantics. Therefore, we utilize click feature to reduce the semantic gap. The second key issue is learning an appropriate distance metric to combine these multimodal features. This paper develops a novel deep multimodal distance metric learning (Deep-MDML) method. A structured ranking model is adopted to utilize both visual and click features in distance metric learning (DML). Specifically, images and their related ranking results are first collected to form the training set. Multimodal features, including click and visual features, are collected with these images. Next, a group of autoencoders is applied to obtain initially a distance metric in different visual spaces, and an MDML method is used to assign optimal weights for different modalities. Next, we conduct alternating optimization to train the ranking model, which is used for the ranking of new queries with click features. Compared with existing image ranking methods, the proposed method adopts a new ranking model to use multimodal features, including click features and visual features in DML. We operated experiments to analyze the proposed Deep-MDML in two benchmark data sets, and the results validate the effects of the method.

Deep Coupled Metric Learning for Cross-Modal Matching.

Cross-modal Deep Metric Learning with Multi-Task Regularization

Large Margin Coupled Feature Learning for Cross-Modal Face Recognition

Deep Compositional Metric Learning

Towards Interpretable Deep Metric Learning with Structural Matching

Cross-Domain Visual Matching via Generalized Similarity Measure and Feature Learning

Multi-scale Matching Networks for Semantic Correspondence

Deep Multi-Modal Image Correspondence Learning

Coupled Deep Learning for Heterogeneous Face Recognition

Two-Stream Convolutional Neural Network For Multimodal Matching

Fully-Trainable Deep Matching

DIML: Deep Interpretable Metric Learning Via Structural Matching.

GSSF: Generalized Structural Sparse Function for Deep Cross-modal Metric Learning

Deep Factorized Metric Learning

Deep cross-modal subspace clustering with Contrastive Neighbour Embedding

Decoupled Cross-Modal Phrase-Attention Network for Image-Sentence Matching

Advanced Multimodal Deep Learning Architecture for Image-Text Matching

Weakly-paired Deep Dictionary Learning for Cross-Modal Retrieval

Multimodal Remote Sensing Image Matching via Learning Features and Attention Mechanism

Deep Multimodal Distance Metric Learning Using Click Constraints for Image Ranking.

A Deep Metric for Multimodal Registration