Abstract:The click feature of an image, defined as a user click count vector based on click data, has been demonstrated to be effective for reducing the semantic gap for image recognition. Unfortunately, most of the traditional image recognition datasets do not contain click data. To address this problem, researchers have begun to develop a click prediction model using assistant datasets containing click information and have adapted this predictor to a common click-free dataset for different tasks. This method can be customized to our problem, but it has two main limitations: 1) the predicted click feature often performs badly in the recognition task since the prediction model is constructed independently of the subsequent recognition problem and 2) transferring the predictor from one dataset to another is challenging due to the large cross-domain diversity. In this paper, we devise a multitask and multidomain deep network with varied modals (MTMDD-VM) to formulate image recognition and click prediction tasks in a unified framework. Datasets with and without click information are integrated in the training. Furthermore, a nonlinear word embedding with a position-sensitive loss function is designed to discover the visual click correlation. We evaluate the proposed method on three public dog breed image datasets, and we utilize the Clickture-Dog dataset as the auxiliary dataset that provides click data. The experimental results show that: 1) the nonlinear word embedding and position-sensitive loss function largely enhance the predicted click feature in the recognition task, realizing a 32% improvement in accuracy; 2) the multitask learning framework improves accuracies in both image recognition and click prediction; and 3) the unified training using the combined dataset with and without click data further improves the performance. Compared with the state-of-the-art methods, the proposed approach not only performs much better in accuracy but also achieves good scalability and one-shot learning ability.

Learning High-level Image Representation for Image Retrieval Via Multi-Task DNN Using Clickthrough Data.

Click-Through Rate Prediction Algorithm Based on Modeling of Implicit High-Order Feature Importance

Bag-Of-Words Based Deep Neural Network For Image Retrieval

Learning Cross Space Mapping Via DNN Using Large Scale Click-Through Logs

Image Recognition by Predicted User Click Feature with Multidomain Multitask Transfer Deep Network

Multi-level Image Representation for Large-Scale Image-Based Instance Retrieval.

Deep Learning for Content-Based Image Retrieval: A Comprehensive Study

Deep Neural Network Boosted Large Scale Image Recognition Using User Click Data.

Learning to Rank Using User Clicks and Visual Features for Image Retrieval

Representative Feature Matching Network for Image Retrieval

Deep Neural Network Retrieval.

Effective Deep Learning-Based Multi-Modal Retrieval

Learning Token-Based Representation for Image Retrieval

Dual Attention Triplet Hashing Network for Image Retrieval.

Deep Click Feature Based Query Merging for Robust Image Recognition.

Deep Cross-Dimensional Attention Hashing for Image Retrieval.

Learning Deep Local Features with Multiple Dynamic Attentions for Large-Scale Image Retrieval.

Multi-level similarity learning for image-text retrieval

Deep Multi-View Enhancement Hashing for Image Retrieval.

Learning of Multimodal Representations With Random Walks on the Click Graph

Large-Scale Remote Sensing Image Retrieval by Deep Hashing Neural Networks