Abstract:Reinforcement learning (RL) has been widely applied in recommendation systems due to its potential in optimizing the long-term engagement of users. From the perspective of RL, recommendation can be formulated as a Markov decision process (MDP), where recommendation system (agent) can interact with users (environment) and acquire feedback (reward signals).However, it is impractical to conduct online interactions with the concern on user experience and implementation complexity, and we can only train RL recommenders with offline datasets containing limited reward signals and state transitions. Therefore, the data sparsity issue of reward signals and state transitions is very severe, while it has long been overlooked by existing RL <a class="link-external link-http" href="http://recommenders.Worse" rel="external noopener nofollow">this http URL</a> still, RL methods learn through the trial-and-error mode, but negative feedback cannot be obtained in implicit feedback recommendation tasks, which aggravates the overestimation problem of offline RL recommender. To address these challenges, we propose a novel RL recommender named model-enhanced contrastive reinforcement learning (MCRL). On the one hand, we learn a value function to estimate the long-term engagement of users, together with a conservative value learning mechanism to alleviate the overestimation <a class="link-external link-http" href="http://problem.On" rel="external noopener nofollow">this http URL</a> the other hand, we construct some positive and negative state-action pairs to model the reward function and state transition function with contrastive learning to exploit the internal structure information of MDP. Experiments demonstrate that the proposed method significantly outperforms existing offline RL and self-supervised RL methods with different representative backbone networks on two real-world datasets.

Towards Off-Policy Reinforcement Learning for Ranking Policies with Human Feedback

Beyond Reward: Offline Preference-guided Policy Optimization

Reinforcement Online Learning to Rank with Unbiased Reward Shaping

Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Application

Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization

Fine-Tuning Language Models with Reward Learning on Policy

ROLeR: Effective Reward Shaping in Offline Reinforcement Learning for Recommender Systems

Optimizing Long-term Value for Auction-Based Recommender Systems via On-Policy Reinforcement Learning

LT2R: Learning to Online Learning to Rank for Web Search

Ranking Policy Learning via Marketplace Expected Value Estimation From Observational Data

An Off-Policy Reinforcement Learning Algorithm Customized for Multi-Task Fusion in Large-Scale Recommender Systems

A General Offline Reinforcement Learning Framework for Interactive Recommendation

Learned Ranking Function: From Short-term Behavior Predictions to Long-term User Satisfaction

Rethinking Reinforcement Learning for Recommendation: A Prompt Perspective

Reinforcing Long-Term Performance in Recommender Systems with User-Oriented Exploration Policy

Optimizing Ranking Algorithm in Recommender System Via Deep Reinforcement Learning

Offline Deep Reinforcement Learning Two-stage Optimization Framework Applied to Recommendation Systems

PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagement

User Retention-oriented Recommendation with Decision Transformer.

Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint

Model-enhanced Contrastive Reinforcement Learning for Sequential Recommendation