Visual Reranking: From Objectives to Strategies

xinmei tian,Dacheng Tao

DOI: https://doi.org/10.1109/MMUL.2011.36

IF: 3.4911

2011-01-01

IEEE Multimedia

Abstract:A study of the development of visual reranking methods can facilitate an understanding of the field, offer a clearer view of what has been achieved, and help overcome emerging obstacles in this area.

What problem does this paper attempt to address?

Learning to rerank images with enhanced spatial verification

Chang Xu,Yangxi Li,Chao Zhou,Chao Xu

DOI: https://doi.org/10.1109/ICIP.2012.6467264

2012-01-01

ICIP

Abstract:Reranking is one of the commonly used schemes to improve the initial ranking performance for content based image retrieval (CBIR). The state-of-the-art reranking methods for CBIR are mainly based on spatial verification and global feature. To mine the complementary properties of different reranking strategies, we combine features representing images from different perspectives with RankSVM to obtain a reranking model to refine the initial ranking list. Besides, compared with traditional spatial verification based methods which measure image similarity only with single inlier's statistical properties, we bind close inlier visual words together to mine more geometric information from images. Through organizing inliers into sequence and computing the relative positions among inliers, we define an efficient similarity measurement with the order consistency between inlier sequences. Experimental results on both Oxford and imageNet datasets demonstrate that our proposed reranking method is effective and promising.
Multimedia search reranking

Tao Mei,Yong Rui,Shipeng Li,Qi Tian

DOI: https://doi.org/10.1145/2536798

IF: 16.6

2014-01-01

ACM Computing Surveys

Abstract:The explosive growth and widespread accessibility of community-contributed media content on the Internet have led to a surge of research activity in multimedia search. Approaches that apply text search techniques for multimedia search have achieved limited success as they entirely ignore visual content as a ranking signal. Multimedia search reranking, which reorders visual documents based on multimodal cues to improve initial text-only searches, has received increasing attention in recent years. Such a problem is challenging because the initial search results often have a great deal of noise. Discovering knowledge or visual patterns from such a noisy ranked list to guide the reranking process is difficult. Numerous techniques have been developed for visual search re-ranking. The purpose of this paper is to categorize and evaluate these algorithms. We also discuss relevant issues such as data collection, evaluation metrics, and benchmarking. We conclude with several promising directions for future research.
Visual search reranking via adaptive particle swarm optimization

Lu Zhang,Tao Mei,Yuan Liu,Dacheng Tao,He-Qin Zhou

DOI: https://doi.org/10.1016/j.patcog.2011.01.016

IF: 8

2011-01-01

Pattern Recognition

Abstract:Visual search reranking involves an optimization process that uses visual content to recover the ''genuine'' ranking list from the helpful but noisy one generated by textual search. This paper presents an evolutionary approach, called Adaptive Particle Swarm Optimization (APSO), for unsupervised visual search reranking. The proposed approach incorporates the visual consistency regularization and the ranking list distance. In addition, to address the problem that existing list distance fails to capture the genuine disagreement between two ranking lists, we propose a numerical ranking list distance. Furthermore, the parameters in APSO are self-tuned adaptively according to the fitness values of the particles to avoid being trapped in local optima. We conduct extensive experiments on automatic search task over TRECVID 2006-2007 benchmarks and show significant and consistent improvements over state-of-the-art works.
Multimedia search reranking: A literature survey

Tao Mei,Yong Rui,Shipeng Li,Qi Tian

DOI: https://doi.org/10.1145/2536798

2014-01-01

Abstract:The explosive growth and widespread accessibility of community-contributed media content on the Internet have led to a surge of research activity in multimedia search. Approaches that apply text search techniques for multimedia search have achieved limited success as they entirely ignore visual content as a ranking signal. Multimedia search reranking, which reorders visual documents based on multimodal cues to improve initial text-only searches, has received increasing attention in recent years. Such a problem is challenging because the initial search results often have a great deal of noise. Discovering knowledge or visual patterns from such a noisy ranked list to guide the reranking process is difficult. Numerous techniques have been developed for visual search re-ranking. The purpose of this paper is to categorize and evaluate these algorithms. We also discuss relevant issues such as data collection, evaluation metrics, and benchmarking. We conclude with several promising directions for future research.
Spatial Consistency Based Selective Reranking for Content Based Object Retrieval.

Tan Xiao,Chao Zhang,Hongbin Zha

2012-01-01

Abstract:Reranking is one of the commonly used methods to improve the initial ranking performance for content based object retrieval. In this paper, we propose a spatial consistency based selective reranking method to boost the performance of traditional reranking. After deriving the query's top results, we measure the spatial consistency degree of each query-result image pair via visual words spatial verification. Then the images which are most consistent to query image are selected as positive examples for current query images, and utilized in pseudo relevance feedback reranking. On the other hand, for queries whose top results are less consistent to the query images, they are excluded from reranking to avoid unnecessary performance deterioration. Experimental results on two datasets demonstrate the effectiveness of our method in object retrieval applications.
A Novel and Adaptive Method for Image Search Reranking

Hong Lu,Guobao Jiang,Zhuohong Cai,Xiangyang Xue

DOI: https://doi.org/10.1007/978-3-642-34595-1_30

2012-01-01

Abstract:In this paper, we propose a novel and adaptive method for image search reranking. We firstly evaluate different visual features based on the results of image classification on object and scene separately. And visual features are chosen adaptively to rerank the initial image search result. For a given query, it can be classified into either object or scene using the trained classifier on text features. Then, low-level visual features are adaptively selected and fused for image search reranking. Experimental results on large scale image dataset of WebQueries demonstrate the efficacy of the proposed method.
Visual reranking with improved image graph

Ziqiong Liu,Shengjin Wang,Liang Zheng,Qi Tian

DOI: https://doi.org/10.1109/ICASSP.2014.6854935

2014-01-01

ICASSP

Abstract:This paper introduces an improved reranking method for the Bag-of-Words (BoW) based image search. Built on [1], a directed image graph robust to outlier distraction is proposed. In our approach, the relevance among images is encoded in the image graph, based on which the initial rank list is refined. Moreover, we show that the rank-level feature fusion can be adopted in this reranking method as well. Taking advantage of the complementary nature of various features, the reranking performance is further enhanced. Particularly, we exploit the reranking method combining the BoW and color information. Experiments on two benchmark datasets demonstrate that our method yields significant improvements and the reranking results are competitive to the state-of-the-art methods.
Optimizing Multimodal Reranking for Web Image Search

Hao Li,Meng Wang,Zhisheng Li,Zheng-Jun Zha,Jialie Shen

DOI: https://doi.org/10.1145/2009916.2010078

2011-01-01

Abstract:In this poster, we introduce a web image search reranking approach with exploring multiple modalities. Diff erent from the conventional methods that build graph with one feature set for reranking, our approach integrates multiple feature sets that describe visual content from different aspects. We simultaneously integrate the learning of relevance scores, the weighting of different feature sets, the distance metric and the scaling for each feature set into a unified scheme. Experimental results on a large data set that contains more than 1,100 queries and 1 million images demonstrate the effectiveness of our approach.
Content-aware Ranking for Visual Search

Bo Geng,Linjun Yang,Chao Xu,Xian-Sheng Hua

DOI: https://doi.org/10.1109/cvpr.2010.5540003

2010-01-01

Abstract:The ranking models of existing image/video search engines are generally based on associated text while the visual content is actually neglected. Imperfect search results frequently appear due to the mismatch between the textual features and the actual visual content. Visual reranking, in which visual information is applied to refine text based search results, has been proven to be effective. However, the improvement brought by visual reranking is limited, and the main reason is that the errors in the text-based results will propagate to the refinement stage. In this paper, we propose a Content-Aware Ranking model based on “learning to rank” framework, in which textual and visual information are simultaneously leveraged in the ranking learning process. We formulate the Content-Aware Ranking based on large margin structured output learning, by modeling the visual information into a regularization term. The direct optimization of the learning problem is nearly infeasible since the number of constraints is huge. The efficient cutting plane algorithm is adopted to learn the model by iteratively adding the most violated constraints. Extensive experimental results on a large-scale dataset collected from a commercial Web image search engine, as well as the TRECVID 2007 video search dataset, demonstrate that the proposed ranking model significantly outperforms the state-of-the-art ranking and reranking methods.
Active reranking for web image search.

Xinmei Tian,Dacheng Tao,Xian-Sheng Hua,Xiuqing Wu

DOI: https://doi.org/10.1109/TIP.2009.2035866

IF: 10.6

2010-01-01

IEEE Transactions on Image Processing

Abstract:Image search reranking methods usually fail to capture the user's intention when the query term is ambiguous. Therefore, reranking with user interactions, or active reranking, is highly demanded to effectively improve the search performance. The essential problem in active reranking is how to target the user's intention. To complete this goal, this paper presents a structural information based sample selection strategy to reduce the user's labeling efforts. Furthermore, to localize the user's intention in the visual feature space, a novel local-global discriminative dimension reduction algorithm is proposed. In this algorithm, a submanifold is learned by transferring the local geometry and the discriminative information from the labelled images to the whole (global) image database. Experiments on both synthetic datasets and a real Web image search dataset demonstrate the effectiveness of the proposed active reranking scheme, including both the structural information based active sample selection strategy and the local-global discriminative dimension reduction algorithm.
Learning to Rank Retargeted Images.

Yang Chen,Yong-Jin Liu,Yu-Kun Lai

DOI: https://doi.org/10.1109/cvpr.2017.504

2017-01-01

Abstract:Image retargeting techniques that adjust images into different sizes have attracted much attention recently. Objective quality assessment (OQA) of image retargeting results is often desired to automatically select the best results. Existing OQA methods output an absolute score for each retargeted image and use these scores to compare different results. Observing that it is challenging even for human subjects to give consistent scores for retargeting results of different source images, in this paper we propose a learning-based OQA method that predicts the ranking of a set of retargeted images with the same source image. We show that this more manageable task helps achieve more consistent prediction to human preference and is sufficient for most application scenarios. To compute the ranking, we propose a simple yet efficient machine learning framework that uses a General Regression Neural Network (GRNN) to model a combination of seven elaborate OQA metrics. We then propose a simple scheme to transform the relative scores output from GRNN into a global ranking. We train our GRNN model using human preference data collected in the elaborate RetargetMe benchmark and evaluate our method based on the subjective study in RetargetMe. Moreover, we introduce a further subjective benchmark to evaluate the generalizability of different OQA methods. Experimental results demonstrate that our method outperforms eight representative OQA methods in ranking prediction and has better generalizability to different datasets.
A Bird's-eye View of Reranking: from List Level to Page Level

Yunjia Xi,Jianghao Lin,Weiwen Liu,Xinyi Dai,Weinan Zhang,Rui Zhang,Ruiming Tang,Yong Yu

DOI: https://doi.org/10.1145/3539597.3570399

2022-01-01

Abstract:Reranking, as the final stage of multi-stage recommender systems, refines the initial lists to maximize the total utility. With the development of multimedia and user interface design, the recommendation page has evolved to a multi-list style. Separately employing traditional list-level reranking methods for different lists overlooks the inter-list interactions and the effect of different page formats, thus yielding suboptimal reranking performance. Moreover, simply applying a shared network for all the lists fails to capture the commonalities and distinctions in user behaviors on different lists. To this end, we propose to draw a bird's-eye view of page-level reranking and design a novel Page-level Attentional Reranking (PAR) model. We introduce a hierarchical dual-side attention module to extract personalized intra- and inter-list interactions. A spatial-scaled attention network is devised to integrate the spatial relationship into pairwise item influences, which explicitly models the page format. The multi-gated mixture-of-experts module is further applied to capture the commonalities and differences of user behaviors between different lists. Extensive experiments on a public dataset and a proprietary dataset show that PAR significantly outperforms existing baseline models.
Visual Reranking through Weakly Supervised Multi-graph Learning

Cheng Deng,Rongrong Ji,Wei Liu,Dacheng Tao,Xinbo Gao

DOI: https://doi.org/10.1109/ICCV.2013.323

2013-01-01

Abstract:Visual reranking has been widely deployed to refine the quality of conventional content-based image retrieval engines. The current trend lies in employing a crowd of retrieved results stemming from multiple feature modalities to boost the overall performance of visual reranking. However, a major challenge pertaining to current reranking methods is how to take full advantage of the complementary property of distinct feature modalities. Given a query image and one feature modality, a regular visual reranking framework treats the top-ranked images as pseudo positive instances which are inevitably noisy, difficult to reveal this complementary property, and thus lead to inferior ranking performance. This paper proposes a novel image reranking approach by introducing a Co-Regularized Multi-Graph Learning (Co-RMGL) framework, in which the intra-graph and inter-graph constraints are simultaneously imposed to encode affinities in a single graph and consistency across different graphs. Moreover, weakly supervised learning driven by image attributes is performed to denoise the pseudo-labeled instances, thereby highlighting the unique strength of individual feature modality. Meanwhile, such learning can yield a few anchors in graphs that vitally enable the alignment and fusion of multiple graphs. As a result, an edge weight matrix learned from the fused graph automatically gives the ordering to the initially retrieved results. We evaluate our approach on four benchmark image retrieval datasets, demonstrating a significant performance gain over the state-of-the-arts.
An Attribute-Assisted Reranking Model for Web Image Search.

Junjie Cai,Zheng-Jun Zha,Meng Wang,Shiliang Zhang,Qi Tian

DOI: https://doi.org/10.1109/tip.2014.2372616

IF: 10.6

2015-01-01

IEEE Transactions on Image Processing

Abstract:Image search reranking is an effective approach to refine the text-based image search result. Most existing reranking approaches are based on low-level visual features. In this paper, we propose to exploit semantic attributes for image search reranking. Based on the classifiers for all the predefined attributes, each image is represented by an attribute feature consisting of the responses from these classifiers. A hypergraph is then used to model the relationship between images by integrating low-level visual features and attribute features. Hypergraph ranking is then performed to order the images. Its basic principle is that visually similar images should have similar ranking scores. In this paper, we propose a visual-attribute joint hypergraph learning approach to simultaneously explore two information sources. A hypergraph is constructed to model the relationship of all images. We conduct experiments on more than 1,000 queries in MSRA-MMV2.0 data set. The experimental results demonstrate the effectiveness of our approach.
A bag-of-objects retrieval model for web image search.

Yang Yang,Linjun Yang,Gangshan Wu,Shipeng Li

DOI: https://doi.org/10.1145/2393347.2393362

2012-01-01

Abstract:ABSTRACTImage search reranking has been an active research topic in recent years to boost the performance of the existing web image search engine which is mostly based on textual metadata of images. Various approaches have been proposed to rerank images for general queries and argue that, they may not necessarily be optimal for queries in specific domain, e.g., object queries, since the reranking algorithms are operated on whole images, instead of the relevant parts of images. In this paper, we propose a novel bag-of-objects retrieval model for image search reranking of object queries. Firstly, we employ a common object discovery algorithm to discover query-relevant objects from the search results returned by text-based image search engine. Then, the query and its result images are represented as a language model on the query relevant object vocabulary, based on which the ranking function can be derived. As the common object discovery is unreliable and may introduce noises, we propose to incorporate the attributes of the discovered objects, e.g., size, position, etc., into the ranking function through a linear model, and the weights on the object attributes can be learned. The experiments on two subsets of Web Queries dataset comprising object queries demonstrate that our approach can significantly outperform the existing reranking methods on object queries.
Leveraging Exemplar and Saliency Model for Image Search Reranking

Hong Lu,Kai Chen,Guobao Jiang,Renzhong Wei,Xiangyang Xue

DOI: https://doi.org/10.1007/978-3-642-34778-8_16

2012-01-01

Abstract:In this paper, we propose to rerank the image retrieval results using a novel method which can be fitted to both objects classes and scenes classes. We first introduce the two methods: Exemplar model and Saliency Map (SM). Exemplar model is a top-down method which considers region of interest (ROI) of images from the same class containing lots of similar discriminative local features. These discriminative local features can be trained as the model of the specific class and to rerank the retrieved images by their similarities with the trained model of the query class. On the other hand, SM is a bottom-up method which uses winner-take-all and inhibition-of-return mechanisms to draw different locations in descending saliency order, and the images can be reranked by their salient scores. In experimental results, we observe that Exemplar Model performs well in object classes and SM performs well in scene classes for these two methods focus on different aspects to rerank images. Then we propose a method named ExSM which combines the advantage of Exemplar model and SM. ExSM inherits the superiority of Exemplar model in object classes and SM in scene classes and outperforms both of them in general.
Multimodal graph-based reranking for web image search.

Meng Wang,Hao Li,Dacheng Tao,Ke Lu,Xindong Wu

DOI: https://doi.org/10.1109/TIP.2012.2207397

IF: 10.6

2012-01-01

IEEE Transactions on Image Processing

Abstract:This paper introduces a web image search reranking approach that explores multiple modalities in a graph-based learning scheme. Different from the conventional methods that usually adopt a single modality or integrate multiple modalities into a long feature vector, our approach can effectively integrate the learning of relevance scores, weights of modalities, and the distance metric and its scaling for each modality into a unified scheme. In this way, the effects of different modalities can be adaptively modulated and better reranking performance can be achieved. We conduct experiments on a large dataset that contains more than 1000 queries and 1 million images to evaluate our approach. Experimental results demonstrate that the proposed reranking approach is more robust than using each individual modality, and it also performs better than many existing methods.
POP: Person Re-identification Post-rank Optimisation

Chunxiao Liu,Loy, C.C.,Shaogang Gong,Guijin Wang

DOI: https://doi.org/10.1109/ICCV.2013.62

2013-01-01

Abstract:Owing to visual ambiguities and disparities, person re-identification methods inevitably produce sub optimal rank-list, which still requires exhaustive human eyeballing to identify the correct target from hundreds of different likely-candidates. Existing re-identification studies focus on improving the ranking performance, but rarely look into the critical problem of optimising the time-consuming and error-prone post-rank visual search at the user end. In this study, we present a novel one-shot Post-rank Optimization (POP) method, which allows a user to quickly refine their search by either "one-shot" or a couple of sparse negative selections during a re-identification process. We conduct systematic behavioural studies to understand user's searching behaviour and show that the proposed method allows correct re-identification to converge 2.6 times faster than the conventional exhaustive search. Importantly, through extensive evaluations we demonstrate that the method is capable of achieving significant improvement over the state-of-the-art distance metric learning based ranking models, even with just "one shot" feedback optimisation, by as much as over 30% performance improvement for rank 1 re-identification on the VIPeR and i-LIDS datasets.
Image Search Reranking with Hierarchical Topic Awareness.

Xinmei Tian,Linjun Yang,Yijuan Lu,Qi Tian,Dacheng Tao

DOI: https://doi.org/10.1109/tcyb.2014.2366740

IF: 11.8

2015-01-01

IEEE Transactions on Cybernetics

Abstract:With much attention from both academia and industrial communities, visual search reranking has recently been proposed to refine image search results obtained from text-based image search engines. Most of the traditional reranking methods cannot capture both relevance and diversity of the search results at the same time. Or they ignore the hierarchical topic structure of search result. Each topic is treated equally and independently. However, in real applications, images returned for certain queries are naturally in hierarchical organization, rather than simple parallel relation. In this paper, a new reranking method "topic-aware reranking (TARerank)" is proposed. TARerank describes the hierarchical topic structure of search results in one model, and seamlessly captures both relevance and diversity of the image search results simultaneously. Through a structured learning framework, relevance and diversity are modeled in TARerank by a set of carefully designed features, and then the model is learned from human-labeled training samples. The learned model is expected to predict reranking results with high relevance and diversity for testing queries. To verify the effectiveness of the proposed method, we collect an image search dataset and conduct comparison experiments on it. The experimental results demonstrate that the proposed TARerank outperforms the existing relevance-based and diversified reranking methods.
Sparse Transfer Learning for Interactive Video Search Reranking

Xinmei Tian,Dacheng Tao,Yong Rui

DOI: https://doi.org/10.1145/2240136.2240139

IF: 4.094

2012-01-01

ACM Transactions on Multimedia Computing Communications and Applications

Abstract:Visual reranking is effective to improve the performance of the text-based video search. However, existing reranking algorithms can only achieve limited improvement because of the well-known semantic gap between low-level visual features and high-level semantic concepts. In this article, we adopt interactive video search reranking to bridge the semantic gap by introducing user's labeling effort. We propose a novel dimension reduction tool, termed sparse transfer learning (STL), to effectively and efficiently encode user's labeling information. STL is particularly designed for interactive video search reranking. Technically, it (a) considers the pair-wise discriminative information to maximally separate labeled query relevant samples from labeled query irrelevant ones, (b) achieves a sparse representation for the subspace to encodes user's intention by applying the elastic net penalty, and (c) propagates user's labeling information from labeled samples to unlabeled samples by using the data distribution knowledge. We conducted extensive experiments on the TRECVID 2005, 2006 and 2007 benchmark datasets and compared STL with popular dimension reduction algorithms. We report superior performance by using the proposed STL-based interactive video search reranking.

Visual Reranking: From Objectives to Strategies

Learning to rerank images with enhanced spatial verification

Multimedia search reranking

Visual search reranking via adaptive particle swarm optimization

Multimedia search reranking: A literature survey

Spatial Consistency Based Selective Reranking for Content Based Object Retrieval.

A Novel and Adaptive Method for Image Search Reranking

Visual reranking with improved image graph

Optimizing Multimodal Reranking for Web Image Search

Content-aware Ranking for Visual Search

Active reranking for web image search.

Learning to Rank Retargeted Images.

A Bird's-eye View of Reranking: from List Level to Page Level

Visual Reranking through Weakly Supervised Multi-graph Learning

An Attribute-Assisted Reranking Model for Web Image Search.

A bag-of-objects retrieval model for web image search.

Leveraging Exemplar and Saliency Model for Image Search Reranking

Multimodal graph-based reranking for web image search.

POP: Person Re-identification Post-rank Optimisation

Image Search Reranking with Hierarchical Topic Awareness.

Sparse Transfer Learning for Interactive Video Search Reranking