Abstract:As the heart of a search engine, the ranking system plays a crucial role in satisfying users' information demands. More recently, neural rankers fine-tuned from pre-trained language models (PLMs) establish state-of-the-art ranking effectiveness. However, it is nontrivial to directly apply these PLM-based rankers to the large-scale web search system due to the following challenging issues: (1) the prohibitively expensive computations of massive neural PLMs, especially for long texts in the web document, prohibit their deployments in an online ranking system that demands extremely low latency; (2) the discrepancy between existing ranking-agnostic pre-training objectives and the ad-hoc retrieval scenarios that demand comprehensive relevance modeling is another main barrier for improving the online ranking system; (3) a real-world search engine typically involves a committee of ranking components, and thus the compatibility of the individually fine-tuned ranking model is critical for a cooperative ranking system. In this work, we contribute a series of successfully applied techniques in tackling these exposed issues when deploying the state-of-the-art Chinese pre-trained language model, i.e., ERNIE, in the online search engine system. We first articulate a novel practice to cost-efficiently summarize the web document and contextualize the resultant summary content with the query using a cheap yet powerful Pyramid-ERNIE architecture. Then we endow an innovative paradigm to finely exploit the large-scale noisy and biased post-click behavioral data for relevance-oriented pre-training. We also propose a human-anchored fine-tuning strategy tailored for the online ranking system, aiming to stabilize the ranking signals across various online components. Extensive offline and online experimental results show that the proposed techniques significantly boost the search engine's performance.

Understanding the Behaviors of BERT in Ranking

An Analysis of BERT in Document Ranking

Investigating the Successes and Failures of BERT for Passage Re-Ranking

Composite Re-Ranking for Efficient Document Search with BERT

Pretrained Transformers for Text Ranking: BERT and Beyond

Dealing with Typos for BERT-based Passage Retrieval and Ranking

A Closer Look at How Fine-tuning Changes BERT

Best Practices for Distilling Large Language Models into BERT for Web Search Ranking

A Pairwise Probe for Understanding BERT Fine-Tuning on Machine Reading Comprehension

Understanding Performance of Long-Document Ranking Models through Comprehensive Evaluation and Leaderboarding

An in-depth analysis of passage-level label transfer for contextual document ranking

Utilizing passage‐level relevance and kernel pooling for enhancing BERT‐based document reranking

Towards Interpreting BERT for Reading Comprehension Based QA

Can Fine-tuning Pre-trained Models Lead to Perfect NLP? A Study of the Generalizability of Relation Extraction.

A Context-Aware BERT Retrieval Framework Utilizing Abstractive Summarization.

A Self-supervised Joint Training Framework for Document Reranking.

Probing Ranking LLMs: Mechanistic Interpretability in Information Retrieval

Pre-trained Language Model based Ranking in Baidu Search

Using Prior Knowledge to Guide BERT's Attention in Semantic Textual Matching Tasks

CAT-BERT: A Context-Aware Transferable BERT Model for Multi-turn Machine Reading Comprehension.

YES SIR!Optimizing Semantic Space of Negatives with Self-Involvement Ranker