Abstract:While China has become the biggest online market in the world with around 1 billion internet users, Baidu runs the world largest Chinese search engine serving more than hundreds of millions of daily active users and responding billions queries per day. To handle the diverse query requests from users at web-scale, Baidu has done tremendous efforts in understanding users' queries, retrieve relevant contents from a pool of trillions of webpages, and rank the most relevant webpages on the top of results. Among these components used in Baidu search, learning to rank (LTR) plays a critical role and we need to timely label an extremely large number of queries together with relevant webpages to train and update the online LTR models. To reduce the costs and time consumption of queries/webpages labeling, we study the problem of Activ Learning to Rank (active LTR) that selects unlabeled queries for annotation and training in this work. Specifically, we first investigate the criterion -- Ranking Entropy (RE) characterizing the entropy of relevant webpages under a query produced by a sequence of online LTR models updated by different checkpoints, using a Query-By-Committee (QBC) method. Then, we explore a new criterion namely Prediction Variances (PV) that measures the variance of prediction results for all relevant webpages under a query. Our empirical studies find that RE may favor low-frequency queries from the pool for labeling while PV prioritizing high-frequency queries more. Finally, we combine these two complementary criteria as the sample selection strategies for active learning. Extensive experiments with comparisons to baseline algorithms show that the proposed approach could train LTR models achieving higher Discounted Cumulative Gain (i.e., the relative improvement {\Delta}DCG4=1.38%) with the same budgeted labeling efforts.

Learning with Both Unlabeled Data and Query Logs for Image Search.

Learning an Image Manifold for Retrieval

Category-Level Regularized Unlabeled-to-Labeled Learning for Semi-supervised Prostate Segmentation with Multi-site Unlabeled Data

Search Task Success Evaluation by Exploiting Multi-View Active Semi-Supervised Learning.

Generalized Manifold-Ranking-Based Image Retrieval

Few-shot learning for remote sensing image retrieval with maml

Learning to Rank Using User Clicks and Visual Features for Image Retrieval

A Simple yet Effective Framework for Active Learning to Rank

Learning with Unlabeled Data and Its Application to Image Retrieval

Enhancing Relevance Feedback in Image Retrieval Using Unlabeled Data

From Document to Image: Learning a Scalable Ranking Model for Content Based Image Retrieval

Exploiting Unlabeled Data In Content-Based Image Retrieval

Meta Learning to Rank for Sparsely Supervised Queries

Visual Reranking through Weakly Supervised Multi-graph Learning

Multi-label Active Learning with Low-Rank Mapping for Image Classification.

Learning a Replacement Model for Query Segmentation with Consistency in Search Logs.

Learning Dual Multi-Scale Manifold Ranking for Semantic Segmentation of High-Resolution Images

A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning

Efficient Manifold Ranking for Image Retrieval

Exploiting Unlabeled Data in CNNs by Self-supervised Learning to Rank

Exploiting Click Constraints and Multi-view Features for Image Re-ranking