Abstract:While China has become the biggest online market in the world with around 1 billion internet users, Baidu runs the world largest Chinese search engine serving more than hundreds of millions of daily active users and responding billions queries per day. To handle the diverse query requests from users at web-scale, Baidu has done tremendous efforts in understanding users' queries, retrieve relevant contents from a pool of trillions of webpages, and rank the most relevant webpages on the top of results. Among these components used in Baidu search, learning to rank (LTR) plays a critical role and we need to timely label an extremely large number of queries together with relevant webpages to train and update the online LTR models. To reduce the costs and time consumption of queries/webpages labeling, we study the problem of Activ Learning to Rank (active LTR) that selects unlabeled queries for annotation and training in this work. Specifically, we first investigate the criterion -- Ranking Entropy (RE) characterizing the entropy of relevant webpages under a query produced by a sequence of online LTR models updated by different checkpoints, using a Query-By-Committee (QBC) method. Then, we explore a new criterion namely Prediction Variances (PV) that measures the variance of prediction results for all relevant webpages under a query. Our empirical studies find that RE may favor low-frequency queries from the pool for labeling while PV prioritizing high-frequency queries more. Finally, we combine these two complementary criteria as the sample selection strategies for active learning. Extensive experiments with comparisons to baseline algorithms show that the proposed approach could train LTR models achieving higher Discounted Cumulative Gain (i.e., the relative improvement {\Delta}DCG4=1.38%) with the same budgeted labeling efforts.

A Comparison and Improvement of Online Learning Algorithms for Sequence Labeling.

Adaptive Client Sampling in Federated Learning via Online Learning with Bandit Feedback

Comparisons of sequence labeling algorithms

Confidence-weighted Online Sequence Labeling Algorithm

Sequential Label Enhancement.

Design Challenges and Misconceptions in Neural Sequence Labeling.

Towards easier and faster sequence labeling for natural language processing: A search-based probabilistic online learning framework (SAPO)

Online Passive-Aggressive Multilabel Classification Algorithms

Labelwise Margin Maximization For Sequence Labeling

A Fast Accurate Two-stage Training Algorithm for L1-regularized CRFs with Heuristic Line Search Strategy.

Labeled Phrase Latent Dirichlet Allocation and Its Online Learning Algorithm

Ordinal Label Distribution Learning

A Simple yet Effective Framework for Active Learning to Rank

An Empirical Study on the Efficacy of Deep Active Learning Techniques

Asynchronous Accelerated Stochastic Gradient Descent.

An Online Inference Algorithm For Labeled Latent Dirichlet Allocation

Learning Label Correlations for Multi-Label Online Passive Aggressive Classification Algorithm

Enhancing Multi-Label Classification Via Dynamic Label-Order Learning

Two-dimensional multilabel active learning with an efficient online adaptation model for image classification.

Online Learning for DNN Training: A Stochastic Block Adaptive Gradient Algorithm