Abstract: Recently, pre-trained language models such as BERT have been applied to document ranking for information retrieval, which first pre-train a general language model on an unlabeled large corpus and then conduct ranking-specific fine-tuning on expert-labeled relevance datasets. Ideally, an IR system would model relevance from a user-system dualism: the user's view and the system's view. User's view judges the relevance based on the activities of "real users" while the system's view focuses on the relevance signals from the system side, e.g., from the experts or algorithms, etc. Inspired by the user-system relevance views and the success of pre-trained language models, in this paper we propose a novel ranking framework called Pre-Rank that takes both user's view and system's view into consideration, under the pre-training and fine-tuning paradigm. Specifically, to model the user's view of relevance, Pre-Rank pre-trains the initial query-document representations based on large-scale user activities data such as the click log. To model the system's view of relevance, Pre-Rank further fine-tunes the model on expert-labeled relevance data. More importantly, the pre-trained representations, are fine-tuned together with handcrafted learning-to-rank features under a wide and deep network architecture. In this way, Pre-Rank can model the relevance by incorporating the relevant knowledge and signals from both real search users and the IR experts. To verify the effectiveness of Pre-Rank, we showed two implementations by using BERT and SetRank as the underlying ranking model, respectively. Experimental results base on three publicly available benchmarks showed that in both of the implementations, Pre-Rank can respectively outperform the underlying ranking models and achieved state-of-the-art performances.

Investigating Weak Supervision in Deep Ranking.

Training Deep Ranking Model with Weak Relevance Labels

Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision

Dense Re-Ranking with Weak Supervision for RDF Dataset Search

Generalized Weak Supervision for Neural Information Retrieval

Learning to Rank with Document Ranks and Scores

Modeling Relevance Ranking under the Pre-training and Fine-tuning Paradigm

Semi-supervised document retrieval

DeepRank: A New Deep Architecture for Relevance Ranking in Information Retrieval

A Deep Look into neural ranking models for information retrieval

Meta Learning to Rank for Sparsely Supervised Queries

Learning to rank based on user relevance feedback

Search Result Reranking with Visual and Structure Information Sources

Learning to Rank with Query-level Semi-supervised Autoencoders

A New Deep Neural Network Based Learning to Rank Method for Information Retrieval

Context-Aware Ranking by Constructing a Virtual Environment for Reinforcement Learning

Ranking with Deep Multi-Objective Learning

Sogou-QCL

Integrating Representation and Interaction for Context-Aware Document Ranking

A Deep Recurrent Survival Model for Unbiased Ranking

Incorporating robustness into web ranking evaluation.