Robust Semantic Sketch Based Specific Image Retrieval

Cailiang Liu,Dong Wang,Xiaobing Liu,Changhu Wang,Lei Zhang,Bo Zhang

DOI: https://doi.org/10.1109/icme.2010.5583065

2010-01-01

Abstract:Specific images refer to images one has a certain episodic memory about, e.g. a picture one has ever seen before. Specific image retrieval is a frequent daily information need and the episodic memory is the key to find a specific image. In this paper, we propose a novel semantic sketch-based interface to incorporate the episodic memory for specific image retrieval. The interface allows a user to specify the semantic category and rough area/color of the objects in his memory. To bridge the semantic gap between the query sketch and database images, in the back end, a sampling method selects exemplars from a reference data set which contains many object instances with user-provided tags and bounding boxes. After that, an exemplar matching algorithm ranks images to retrieve the target image to match the user's memory. In practice, we have observed that query sketches are usually error prone. That is, the position or the color of an object may not be accurate. Mean while, the annotations in the reference data set are also noisy. Thus, the search algorithm has to handle two kinds of errors :1) reference data set label noise; 2) user sketch error such as position or scale. For the former, we propose a robust sampling method. For the latter, we derive an efficient spatial reranking algorithm to tolerate in accurate user sketches. Detailed experimental results on the Label Me data set show that the proposed approach is robust to both kinds of errors.