Abstract:Learning-to-rank has been intensively studied and has shown significantly increasing values in a wide range of domains, such as web search, recommender systems, dialogue systems, machine translation, and even computational biology, to name a few. In light of recent advances in neural networks, there has been a strong and continuing interest in exploring how to deploy popular techniques, such as reinforcement learning and adversarial learning, to solve ranking problems. However, armed with the aforesaid popular techniques, most studies tend to show how effective a new method is. A comprehensive comparison between techniques and an in-depth analysis of their deficiencies are somehow overlooked. This paper is motivated by the observation that recent ranking methods based on either reinforcement learning or adversarial learning boil down to policy-gradient-based optimization. Based on the widely used benchmark collections with complete information (where relevance labels are known for all items), such as MSLRWEB30K and Yahoo-Set1, we thoroughly investigate the extent to which policy-gradient-based ranking methods are effective. On one hand, we analytically identify the pitfalls of policy-gradient-based ranking. On the other hand, we experimentally compare a wide range of representative methods. The experimental results echo our analysis and show that policy-gradient-based ranking methods are, by a large margin, inferior to many conventional ranking methods. Regardless of whether we use reinforcement learning or adversarial learning, the failures are largely attributable to the gradient estimation based on sampled rankings, which significantly diverge from ideal rankings. In particular, the larger the number of documents per query and the more fine-grained the ground-truth labels, the greater the impact policy-gradient-based ranking suffers. Careful examination of this weakness is highly recommended for developing enhanced methods based on policy gradient.

Reinforcement Learning to Rank with Pairwise Policy Gradient

Diagnostic Evaluation of Policy-Gradient-Based Ranking

Policy-Gradient Training of Language Models for Ranking

Reinforcement Learning to Rank with Markov Decision Process

Learning to rank: from pairwise approach to listwise approach

Multi Page Search with Reinforcement Learning to Rank

Learning to Re-Rank Web Search Results with Multiple Pairwise Features

RLPS: A Reinforcement Learning–Based Framework for Personalized Search

PL-ranking

Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Application

An Attention-Based Interactive Learning-to-Rank Model for Document Retrieval

An Overview of Learning to Rank for Information Retrieval

Online Learning to Rank in a Listwise Approach for Information Retrieval

Mixed Policy Gradient: off-policy reinforcement learning driven jointly by data and model

Towards Off-Policy Reinforcement Learning for Ranking Policies with Human Feedback

Learning to Collaborate: Multi-Scenario Ranking Via Multi-Agent Reinforcement Learning.

RLPer: A Reinforcement Learning Model for Personalized Search.

Pareto Pairwise Ranking for Fairness Enhancement of Recommender Systems

SetRank: Learning a Permutation-Invariant Ranking Model for Information Retrieval

Sequential Search with Off-Policy Reinforcement Learning

Policy Gradient Methods for Risk-Sensitive Distributional Reinforcement Learning with Provable Convergence