Abstract:The unbiased learning to rank (ULTR) problem has been greatly advanced by recent deep learning techniques and well-designed debias algorithms. However, promising results on the existing benchmark datasets may not be extended to the practical scenario due to the following disadvantages observed from those popular benchmark datasets: (1) outdated semantic feature extraction where state-of-the-art large scale pre-trained language models like BERT cannot be exploited due to the missing of the original text;(2) incomplete display features for in-depth study of ULTR, e.g., missing the displayed abstract of documents for analyzing the click necessary bias; (3) lacking real-world user feedback, leading to the prevalence of synthetic datasets in the empirical study. To overcome the above disadvantages, we introduce the Baidu-ULTR dataset. It involves randomly sampled 1.2 billion searching sessions and 7,008 expert annotated queries, which is orders of magnitude larger than the existing ones. Baidu-ULTR provides:(1) the original semantic feature and a pre-trained language model for easy usage; (2) sufficient display information such as position, displayed height, and displayed abstract, enabling the comprehensive study of different biases with advanced techniques such as causal discovery and meta-learning; and (3) rich user feedback on search result pages (SERPs) like dwelling time, allowing for user engagement optimization and promoting the exploration of multi-task learning in ULTR. In this paper, we present the design principle of Baidu-ULTR and the performance of benchmark ULTR algorithms on this new data resource, favoring the exploration of ranking for long-tail queries and pre-training tasks for ranking. The Baidu-ULTR dataset and corresponding baseline implementation are available at <a class="link-external link-https" href="https://github.com/ChuXiaokai/baidu_ultr_dataset" rel="external noopener nofollow">this https URL</a>.

Sogou-ST: A New Dataset with Large-scale Refined Real-world Web Search Sessions

Tiangong-St: A New Dataset With Large-Scale Refined Real-World Web Search Sessions

SogouQ: The First Large-Scale Test Collection with Click Streams Used in a Shared-Task Evaluation

SogouT-16

Sogou-QCL

MS MARCO Web Search: a Large-scale Information-rich Web Dataset with Millions of Real Click Labels

Towards More Usable Dataset Search: From Query Characterization to Snippet Generation

ConvSearch: A Open-Domain Conversational Search Behavior Dataset

A Large-Scale Rich Context Query and Recommendation Dataset in Online Knowledge-Sharing

Event-driven Real-time Retrieval in Web Search

Still Haven't Found What You're Looking For -- Detecting the Intent of Web Search Missions from User Interaction Features

A Large Scale Search Dataset for Unbiased Learning to Rank

Behavior-based evaluation of session satisfaction

Investigating Cognitive Effects in Session-level Search User Satisfaction

JDsearch: A Personalized Product Search Dataset with Real Queries and Full Interactions

Investigating Session Search Behavior with Knowledge Graphs

Towards Better Understanding of User Satisfaction in Open-Domain Conversational Search

KuaiSAR: A Unified Search And Recommendation Dataset

Incorporating Query Reformulating Behavior into Web Search Evaluation

BASES: Large-scale Web Search User Simulation with Large Language Model based Agents

Improving Search Snippets in Context-Aware Web Search Scenarios.