Abstract:Learning-to-rank has been intensively studied and has shown significantly increasing values in a wide range of domains, such as web search, recommender systems, dialogue systems, machine translation, and even computational biology, to name a few. In light of recent advances in neural networks, there has been a strong and continuing interest in exploring how to deploy popular techniques, such as reinforcement learning and adversarial learning, to solve ranking problems. However, armed with the aforesaid popular techniques, most studies tend to show how effective a new method is. A comprehensive comparison between techniques and an in-depth analysis of their deficiencies are somehow overlooked. This paper is motivated by the observation that recent ranking methods based on either reinforcement learning or adversarial learning boil down to policy-gradient-based optimization. Based on the widely used benchmark collections with complete information (where relevance labels are known for all items), such as MSLRWEB30K and Yahoo-Set1, we thoroughly investigate the extent to which policy-gradient-based ranking methods are effective. On one hand, we analytically identify the pitfalls of policy-gradient-based ranking. On the other hand, we experimentally compare a wide range of representative methods. The experimental results echo our analysis and show that policy-gradient-based ranking methods are, by a large margin, inferior to many conventional ranking methods. Regardless of whether we use reinforcement learning or adversarial learning, the failures are largely attributable to the gradient estimation based on sampled rankings, which significantly diverge from ideal rankings. In particular, the larger the number of documents per query and the more fine-grained the ground-truth labels, the greater the impact policy-gradient-based ranking suffers. Careful examination of this weakness is highly recommended for developing enhanced methods based on policy gradient.

Reinforcement Learning to Rank with Markov Decision Process

Multi Page Search with Reinforcement Learning to Rank

Reinforcement Learning to Rank with Pairwise Policy Gradient

Learning to Rank with Document Ranks and Scores

Adapting Markov Decision Process for Search Result Diversification

An Empirical Perspective on Learning-to-rank.

Inverse Reinforcement Learning with Multiple Ranked Experts

Context-Aware Ranking by Constructing a Virtual Environment for Reinforcement Learning

Diagnostic Evaluation of Policy-Gradient-Based Ranking

Beyond Probability Ranking Principle: Modeling the Dependencies among Documents

Towards Off-Policy Reinforcement Learning for Ranking Policies with Human Feedback

Policy-Gradient Training of Language Models for Ranking

An Overview of Learning to Rank for Information Retrieval

An Attention-Based Interactive Learning-to-Rank Model for Document Retrieval

Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Application

Top-k learning to rank: labeling, ranking and evaluation.

RLPer: A Reinforcement Learning Model for Personalized Search.

Learning a Deep Listwise Context Model for Ranking Refinement

Ranking with Deep Multi-Objective Learning

PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations

DeepRank: Learning to Rank with Neural Networks for Recommendation.