Oracle in Image Search

NieLiqiang,WangMeng,ZhaZheng-Jun,ChuaTat-Seng

IF: 4.657

2012-01-01

ACM Transactions on Information Systems

Abstract:This article studies a novel problem in image search. Given a text query and the image ranking list returned by an image search system, we propose an approach to automatically predict the search pe...

What problem does this paper attempt to address?

Oracle in Image Search

Liqiang Nie,Meng Wang,Zheng-Jun Zha,Tat-Seng Chua

DOI: https://doi.org/10.1145/2180868.2180875

IF: 4.657

2012-01-01

ACM Transactions on Information Systems

Abstract:This article studies a novel problem in image search. Given a text query and the image ranking list returned by an image search system, we propose an approach to automatically predict the search performance. We demonstrate that, in order to estimate the mathematical expectations of Average Precision (AP) and Normalized Discounted Cumulative Gain (NDCG), we only need to predict the relevance probability of each image. We accomplish the task with a query-adaptive graph-based learning based on the images’ ranking order and visual content. We validate our approach with a large-scale dataset that contains the image search results of 1,165 queries from 4 popular image search engines. Empirical studies demonstrate that our approach is able to generate predictions that are highly correlated with the real search performance. Based on the proposed image search performance prediction scheme, we introduce three applications: image metasearch, multilingual image search, and Boolean image search. Comprehensive experiments are conducted to validate our approach.
Oracle in Image Search: A Content-Based Approach to Performance Prediction

Liqiang Nie,Meng Wang,Zheng-Jun Zha,Tat-Seng Chua

DOI: https://doi.org/10.1145/2180868.2180875

2012-01-01

Abstract:This article studies a novel problem in image search. Given a text query and the image ranking list returned by an image search system, we propose an approach to automatically predict the search performance. We demonstrate that, in order to estimate the mathematical expectations of Average Precision (AP) and Normalized Discounted Cumulative Gain (NDCG), we only need to predict the relevance probability of each image. We accomplish the task with a query-adaptive graph-based learning based on the images’ ranking order and visual content. We validate our approach with a large-scale dataset that contains the image search results of 1,165 queries from 4 popular image search engines. Empirical studies demonstrate that our approach is able to generate predictions that are highly correlated with the real search performance. Based on the proposed image search performance prediction scheme, we introduce three applications: image metasearch, multilingual image search, and Boolean image search. Comprehensive experiments are conducted to validate our approach.
Improve Ranking by Using Image Information

Qing Yu,Shuming Shi,Zhiwei Li,Ji-Rong Wen,Wei-Ying Ma

DOI: https://doi.org/10.1007/978-3-540-71496-5_62

2007-01-01

Abstract:This paper explores the feasibility of including image information embedded in Web pages in relevance computation to improve search performance. In determining the ranking of Web pages against a given query, most (if not all) modern Web search engines consider two kinds of factors: text information (including title, URL, body text, anchor text, etc) and static ranking (e.g. PageRank [1]). Although images have been widely used to help represent Web pages and carry valuable information, little work has been done to take advantage of them in computing the relevance score of a Web page given a query. We propose, in this paper, a framework to contain image information in ranking functions. Preliminary experimental results show that, when image information is used properly, ranking results can be improved.
Fine-Grained Image Search

Lingxi Xie,Jingdong Wang,Bo Zhang,Qi Tian

DOI: https://doi.org/10.1109/tmm.2015.2408566

IF: 7.3

2015-01-01

IEEE Transactions on Multimedia

Abstract:Large-scale image search has been attracting lots of attention from both academic and commercial fields. The conventional bag-of-visual-words (BoVW) model with inverted index is verified efficient at retrieving near-duplicate images, but it is less capable of discovering fine-grained concepts in the query and returning semantically matched search results. In this paper, we suggest that instance search should return not only near-duplicate images, but also fine-grained results, which is usually the actual intention of a user. We propose a new and interesting problem named fine-grained image search, which means that we prefer those images containing the same fine-grained concept with the query. We formulate the problem by constructing a hierarchical database and defining an evaluation method. We thereafter introduce a baseline system using fine-grained classification scores to represent and co-index images so that the semantic attributes are better incorporated in the online querying stage. Large-scale experiments reveal that promising search results are achieved with reasonable time and memory consumption. We hope this paper will be the foundation for future work on image search. We also expect more follow-up efforts along this research topic and look forward to commercial fine-grained image search engines.
Exploiting image contents in web search

Zhi-Hua Zhou,Hong-Bin Dai

2007-01-01

Abstract:Web search is a challenging task. Previous research mainly exploits texts on the Web pages or link information between the pages, while multimedia information is largely ignored. This paper proposes a new framework for Web search, which exploits image contents to help improve the search performance. In this framework, candidate images are retrieved at first by considering their associated text information. Then, images related to the query are identified by analyzing the density of the visual feature space. After that, an image-based rank of the Web pages is generated, which is combined with the traditional keyword-based search result to produce the final search result. Experiments demonstrate the promise of the proposed framework.
Exploration of Image Search Results Quality Assessment

Xinmei Tian,Yijuan Lu,Nate Stender,Linjun Yang,Dacheng Tao

DOI: https://doi.org/10.1109/tbdata.2015.2497710

2015-01-01

IEEE Transactions on Big Data

Abstract:Image retrieval plays an increasingly important role in our daily lives. There are many factors which affect the quality of image search results, including chosen search algorithms, ranking functions, and indexing features. Applying different settings for these factors generates search result lists with varying levels of quality. However, no setting can always perform optimally for all queries. Therefore, given a set of search result lists generated by different settings, it is crucial to automatically determine which result list is the best in order to present it to users. This paper aims to solve this problem and makes four main innovations. First, a preference learning model is proposed to quantitatively study and formulate the best image search result list identification problem. Second, a set of valuable preference learning related features is proposed by exploring the visual characters of returned images. Third, a query-dependent preference learning model is further designed for building a more precise and query-specific model. Fourth, the proposed approach has been tested on a variety of applications including re-ranking ability assessment, optimal search engine selection, and synonymous query suggestion. Extensive experimental results on three image search datasets demonstrate the effectiveness and promising potential of the proposed method.
Visual query suggestion: Towards capturing user intent in internet image search

Zheng-Jun Zha,Linjun Yang,Tao Mei,Meng Wang,Zengfu Wang,Tat-Seng Chua,Xian-Sheng Hua

DOI: https://doi.org/10.1145/1823746.1823747

2010-01-01

Abstract:Query suggestion is an effective approach to bridge the Intention Gap between the users' search intents and queries. Most existing search engines are able to automatically suggest a list of textual query terms based on users' current query input, which can be called Textual Query Suggestion. This article proposes a new query suggestion scheme named Visual Query Suggestion (VQS) which is dedicated to image search. VQS provides a more effective query interface to help users to precisely express their search intents by joint text and image suggestions. When a user submits a textual query, VQS first provides a list of suggestions, each containing a keyword and a collection of representative images in a dropdown menu. Once the user selects one of the suggestions, the corresponding keyword will be added to complement the initial query as the new textual query, while the image collection will be used as the visual query to further represent the search intent. VQS then performs image search based on the new textual query using text search techniques, as well as content-based visual retrieval to refine the search results by using the corresponding images as query examples. We compare VQS against three popular image search engines, and show that VQS outperforms these engines in terms of both the quality of query suggestion and the search performance.
Object-Based Image Retrieval With Attention Analysis And Spatial Re-Ranking

Ke Gao,Shouxun Lin,Yongdong Zhang,Sheng Tang

DOI: https://doi.org/10.1007/978-0-387-87685-6_16

2008-01-01

Abstract:In this paper, a new method is proposed for object-based image retrieval. The user supplies a query object by selecting a region from a query image, and the system returns a ranked list of images that contain the same object, retrieved from a large image database. The main outcomes of this research are as follows: (1) An novel object-based image retrieval framework that integrates effective pre-treatment and re-ranking is presented, (2) a new feature filtration method based on attention analysis is proposed for pre-treatment, (3) to further improve object retrieval precision, we add an efficient spatial configuration model to re-rank the primary retrieval result using Bag of Word method. Experimental results demonstrate the effectiveness of our method.
A Novel and Adaptive Method for Image Search Reranking

Hong Lu,Guobao Jiang,Zhuohong Cai,Xiangyang Xue

DOI: https://doi.org/10.1007/978-3-642-34595-1_30

2012-01-01

Abstract:In this paper, we propose a novel and adaptive method for image search reranking. We firstly evaluate different visual features based on the results of image classification on object and scene separately. And visual features are chosen adaptively to rerank the initial image search result. For a given query, it can be classified into either object or scene using the trained classifier on text features. Then, low-level visual features are adaptively selected and fused for image search reranking. Experimental results on large scale image dataset of WebQueries demonstrate the efficacy of the proposed method.
An Adaptive Query Prototype Modeling Method for Image Search Reranking

Hong Lu,Guobao Jiang,Bohong Yang,Xiangyang Xue

DOI: https://doi.org/10.1109/iccvw.2013.52

2013-01-01

Abstract:As more and more images with user free tags are appearing on the Internet, image search reranking has received considerable attention to help users obtain images relevant to the query. In this paper, we propose to rerank the initial image search results using an adaptive online query modeling method based on local features. For a query and its initial rank list, we construct a visual word dictionary using the randomly selected images in the initial rank list. Then the top N images are incrementally used to evaluate the importance/score of the words for the query. The words with higher scores are selected to model the query. The relevance scores of the words to the query are also kept. Then the model is used to measure the relevance of each image in the image list to the query and rerank the image list according to the measured relevance. This method can obtain optimal local prototype for a query from the top N images (N is not large) without considering many images. Also, the relevances of the top N images to the query are not considered equally by weighting according to their positions in the initial image list. Using this model, the influences from some wrongly returned images at the top of the initial rank list can be filtered out. This method is also query independent and can be used to any other query. Experimental results on a large scale image dataset of Web Queries demonstrate the efficacy of the proposed method.
Predicting The Effectiveness Of Queries For Visual Search

Bing Li,Ling-Yu Duan,Yiming Chen,Rongrong Ji,Wen Gao

DOI: https://doi.org/10.1109/ICASSP.2012.6288389

2012-01-01

ICASSP

Abstract:Poor retrieval performance significantly degenerates users' experience of visual search, especially in mobile search. Ideally, users would like to be alerted when bad queries are present, which helps eliminate latency as well as waste of bandwidth, especially in 3G wireless environment. In this paper, we propose a visual query performance prediction (v-QPP) approach to predict the retrieval effectiveness. We employ latent dirichlet allocation (LDA) to derive latent topics from image database. From the collection statistics, we model the query's specificity based on topics. High specificity helps a retrieval system to derive user's search intent exactly. Moreover, as low discriminative content is difficult to search in terms of distinguishing relevant images from irrelevant one, we propose a topics based inverse concept frequency (t-ICF) model to deal with specific queries but difficult to discriminate in the reference database. Comparison experiments over MPEG CDVS benchmarking datasets have shown our method significantly outperforms existing approaches in document retrieval.
Efficient Indexing for Large Scale Visual Search

Xiao Zhang,Zhiwei Li,Lei Zhang,Wei-Ying Ma,Heung-Yeung Shum

DOI: https://doi.org/10.1109/iccv.2009.5459354

2009-01-01

Abstract:With the popularity of “bag of visual terms” representations of images, many text indexing techniques have been applied in large-scale image retrieval systems. However, due to a fundamental difference between an image query (e.g. 1500 visual terms) and a text query (e.g. 3-5 terms), the usages of some text indexing techniques, e.g. inverted list, are misleading. In this work, we develop a novel indexing technique for this problem. The basic idea is to decompose a document-like representation of an image into two components, one for dimension reduction and the other for residual information preservation. The computing of similarity of two images can be transferred to measuring similarities of their components. The decomposition has two major merits: (1) these components have good properties which enable them to be efficiently indexed and retrieved; (2) The decomposition has better generalization ability than other dimension reduction algorithms. The decomposition can be achieved by either a graphical model or a matrix factorization approach. Theoretic analysis and extensive experiments over a 2.3 million image database show that this framework is scalable to index large scale image database to support fast and accurate visual search.
Learning to Combine Ad-hoc Ranking Functions for Image Retrieval

Yangxi Li,Bo Geng,Chao Zhou,Chao Xu

DOI: https://doi.org/10.1109/icdmw.2011.110

2011-01-01

Abstract:Along with the success of "bag of visual words" scheme in content based image retrieval (CBIR), various technologies in text information retrieval realm have been transferred into image retrieval system and obtain promising performance. However, how to select the suitable ranking technology, such as ranking model, for a specific image database is still an open question. Because most ranking models are data-dependent, it is hard to find an optimal model for all the applications. In this paper, we propose to resolve this problem for CBIR with the learning to rank approach which has been widely utilized in text retrieval. Specifically, we consider several well performed ad-hoc ranking models and use their ranking scores to construct the ranking features for the Ranking SVM framework. To best preserve the spatial structures existed in the visual words of image, we split the image into different size blocks, and design the ranking features with a pyramid approach from large blocks to small blocks. Experimental results on both Oxford and Image Net databases demonstrate the effectiveness of proposed method compared with the performance that individual ranking model is adopted. Moreover, the proposed method brings little computational burden to the system and the efficiency analysis proves its scalability.
Ranking consistency for image matching and object retrieval

Yanzhi Chen,Xi Li,Anthony Dick,Rhys Hill

DOI: https://doi.org/10.1016/j.patcog.2013.09.011

IF: 8

2014-01-01

Pattern Recognition

Abstract:The goal of object retrieval is to rank a set of images by the similarity of their contents to those of a query image. However, it is difficult to measure image content similarity due to visual changes caused by varying viewpoint and environment. In this paper, we propose a simple, efficient method to more effectively measure content similarity from image measurements. Our method is based on the ranking information available from existing retrieval systems. We observe that images within the set which, when used as queries, yield similar ranking lists are likely to be relevant to each other and vice versa. In our method, ranking consistency is used as a verification method to efficiently refine an existing ranking list, in much the same fashion that spatial verification is employed. The efficiency of our method is achieved by a list-wise min-Hash scheme, which allows rapid calculation of an approximate similarity ranking. Experimental results demonstrate the effectiveness of the proposed framework and its applications.
Efficient Indexing for Large-Scale Image Search

Shiliang Zhang

DOI: https://doi.org/10.1145/2808492.2808578

2015-01-01

Abstract:In content-based image retrieval, inverted indexes allow fast access to database images and summarize all knowledge about the database. Indexing multiple clues of image contents allows retrieval algorithms search for relevant images from different perspectives, which is appealing to deliver satisfactory user experiences. However, when incorporating diverse image features during online retrieval, it is challenging to ensure retrieval efficiency and scalability. Besides that, most of current image indexing systems for retrieval view database as a set of individual images. It limits the flexibility of the retrieval framework to conduct sophisticated cross image analysis, resulting in higher memory consumption and sub-optimal retrieval accuracy. To conquer these two issues and achieve highly efficient indexing system, we propose a semantic-aware co-indexing algorithm to jointly embed multiple cues into the inverted indexes and a cross indexing strategy to group highly relevant image together. In this talk, these two algorithms will be introduced. A real-time large-scale image search system built on these algorithms will also be presented.
Image search by graph-based label propagation with image representation from DNN.

Yingwei Pan,Ting Yao,Kuiyuan Yang,Houqiang Li,Chong-Wah Ngo,Jingdong Wang,Tao Mei

DOI: https://doi.org/10.1145/2502081.2508128

2013-01-01

Abstract:Our objective is to estimate the relevance of an image to a query for image search purposes. We address two limitations of the existing image search engines in this paper. First, there is no straightforward way of bridging the gap between semantic textual queries as well as users' search intents and image visual content. Image search engines therefore primarily rely on static and textual features. Visual features are mainly used to identify potentially useful recurrent patterns or relevant training examples for complementing search by image reranking. Second, image rankers are trained on query-image pairs labeled by human experts, making the annotation intellectually expensive and time-consuming. Furthermore, the labels may be subjective when the queries are ambiguous, resulting in difficulty in predicting the search intention. We demonstrate that the aforementioned two problems can be mitigated by exploring the use of click-through data, which can be viewed as the footprints of user searching behavior, as an effective means of understanding query. The correspondences between an image and a query are determined by whether the image was searched and clicked by users under the query in a commercial image search engine. We therefore hypothesize that the image click counts in response to a query are as their relevance indications. For each new image, our proposed graph-based label propagation algorithm employs neighborhood graph search to find the nearest neighbors on an image similarity graph built up with visual representations from deep neural networks and further aggregates their clicked queries/click counts to get the labels of the new image. We conduct experiments on MSR-Bing Grand Challenge and the results show consistent performance gain over various baselines. In addition, the proposed approach is very efficient, completing annotation of each query-image pair within just 15 milliseconds on a regular PC.
Layout-Expectation-Based Model for Image Search Re-Ranking

Bin Jin,Weiyao Lin,Jianxin Wu,Tianhao Wu,Jun Huang,Chongyang Zhang

DOI: https://doi.org/10.1109/icmew.2012.28

2012-01-01

Abstract:In this paper, a new algorithm is proposed for image re-ranking in web search applications. The proposed algorithm introduces a new layout expectation model for improving the image search results. The motivation for using the expectation model is that users may often have potential expectations about the desired image during the search process. By including the layout expectation model to describe users' expectation on image layouts, the re-ranked search results can become more satisfactory to users. Experimental results demonstrate that our proposed algorithm can significantly improve the re-ranking precision compared with the state-of-the-art algorithms.
Social-oriented Visual Image Search

Shaowei Liu,Peng Cui,Huanbo Luan,Wenwu Zhu,Shiqiang Yang,Qi Tian

DOI: https://doi.org/10.1016/j.cviu.2013.06.011

IF: 4.886

2014-01-01

Computer Vision and Image Understanding

Abstract:Many research have been focusing on how to match the textual query with visual images and their surrounding texts or tags for Web image search. The returned results are often unsatisfactory due to their deviation from user intentions, particularly for queries with heterogeneous concepts (such as “apple”, “jaguar”) or general (non-specific) concepts (such as “landscape”, “hotel”). In this paper, we exploit social data from social media platforms to assist image search engines, aiming to improve the relevance between returned images and user intentions (i.e., social relevance). Facing the challenges of social data sparseness, the tradeoff between social relevance and visual relevance, and the complex social and visual factors, we propose a community-specific Social-Visual Ranking (SVR) algorithm to rerank the Web images returned by current image search engines. The SVR algorithm is implemented by PageRank over a hybrid image link graph, which is the combination of an image social-link graph and an image visual-link graph. By conducting extensive experiments, we demonstrated the importance of both visual factors and social factors, and the advantages of social-visual ranking algorithm for Web image search.
A comprehensive study on learning to rank for content-based image retrieval

Li Yangxi,Zhou Chao,Geng Bo,Xu Chao,Liu Hong

DOI: https://doi.org/10.1016/j.sigpro.2012.06.012

IF: 4.729

2013-01-01

Signal Processing

Abstract:Recently, various learning to rank approaches have been proposed in the information retrieval realm, with their promising performance in general document and web page retrieval applications. Based on these achievements, in this paper, we investigate and discuss whether learning to rank approaches can be adapted to content-based image retrieval (CBIR). Given the complex structure of image representation, it is also challenging how to design visual features for learning to rank algorithms that not only scale up well, but also model various visual modalities and the spatial distributions of local features. We answer this question by introducing some scalable visual-based ranking features for learning to rank. Specifically, we firstly adopt several well performed ad hoc ranking models to generate the bag-of-visual-words-based ranking features. Besides, images are divided into different salient regions and spatial blocks, respectively, and ranking features are extracted from each region and block. Finally, image global features-based similarities are also concatenated with the existing ranking features. Extensive experiments with three state-of-the-art learning to rank algorithms are performed over four popular image retrieval databases, together with some insightful conclusions to facilitate the adaptation of learning to rank approaches to CBIR. © 2012 Elsevier B.V.
A bag-of-objects retrieval model for web image search.

Yang Yang,Linjun Yang,Gangshan Wu,Shipeng Li

DOI: https://doi.org/10.1145/2393347.2393362

2012-01-01

Abstract:ABSTRACTImage search reranking has been an active research topic in recent years to boost the performance of the existing web image search engine which is mostly based on textual metadata of images. Various approaches have been proposed to rerank images for general queries and argue that, they may not necessarily be optimal for queries in specific domain, e.g., object queries, since the reranking algorithms are operated on whole images, instead of the relevant parts of images. In this paper, we propose a novel bag-of-objects retrieval model for image search reranking of object queries. Firstly, we employ a common object discovery algorithm to discover query-relevant objects from the search results returned by text-based image search engine. Then, the query and its result images are represented as a language model on the query relevant object vocabulary, based on which the ranking function can be derived. As the common object discovery is unreliable and may introduce noises, we propose to incorporate the attributes of the discovered objects, e.g., size, position, etc., into the ranking function through a linear model, and the weights on the object attributes can be learned. The experiments on two subsets of Web Queries dataset comprising object queries demonstrate that our approach can significantly outperform the existing reranking methods on object queries.

Oracle in Image Search

Oracle in Image Search

Oracle in Image Search: A Content-Based Approach to Performance Prediction

Improve Ranking by Using Image Information

Fine-Grained Image Search

Exploiting image contents in web search

Exploration of Image Search Results Quality Assessment

Visual query suggestion: Towards capturing user intent in internet image search

Object-Based Image Retrieval With Attention Analysis And Spatial Re-Ranking

A Novel and Adaptive Method for Image Search Reranking

An Adaptive Query Prototype Modeling Method for Image Search Reranking

Predicting The Effectiveness Of Queries For Visual Search

Efficient Indexing for Large Scale Visual Search

Learning to Combine Ad-hoc Ranking Functions for Image Retrieval

Ranking consistency for image matching and object retrieval

Efficient Indexing for Large-Scale Image Search

Image search by graph-based label propagation with image representation from DNN.

Layout-Expectation-Based Model for Image Search Re-Ranking

Social-oriented Visual Image Search

A comprehensive study on learning to rank for content-based image retrieval

A bag-of-objects retrieval model for web image search.