Abstract:Reinforcement learning (RL) has been widely applied in recommendation systems due to its potential in optimizing the long-term engagement of users. From the perspective of RL, recommendation can be formulated as a Markov decision process (MDP), where recommendation system (agent) can interact with users (environment) and acquire feedback (reward signals).However, it is impractical to conduct online interactions with the concern on user experience and implementation complexity, and we can only train RL recommenders with offline datasets containing limited reward signals and state transitions. Therefore, the data sparsity issue of reward signals and state transitions is very severe, while it has long been overlooked by existing RL <a class="link-external link-http" href="http://recommenders.Worse" rel="external noopener nofollow">this http URL</a> still, RL methods learn through the trial-and-error mode, but negative feedback cannot be obtained in implicit feedback recommendation tasks, which aggravates the overestimation problem of offline RL recommender. To address these challenges, we propose a novel RL recommender named model-enhanced contrastive reinforcement learning (MCRL). On the one hand, we learn a value function to estimate the long-term engagement of users, together with a conservative value learning mechanism to alleviate the overestimation <a class="link-external link-http" href="http://problem.On" rel="external noopener nofollow">this http URL</a> the other hand, we construct some positive and negative state-action pairs to model the reward function and state transition function with contrastive learning to exploit the internal structure information of MDP. Experiments demonstrate that the proposed method significantly outperforms existing offline RL and self-supervised RL methods with different representative backbone networks on two real-world datasets.

A Reinforcement Learning Framework for Eliciting High Quality Information

STRAPPER: Preference-based Reinforcement Learning via Self-training Augmentation and Peer Regularization

Reinforcement Learning From Imperfect Corrective Actions And Proxy Rewards

Prototypical Reward Network for Data-Efficient RLHF

Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization

A Reinforcement Learning Framework for Explainable Recommendation

How to Evaluate Reward Models for RLHF

Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble

PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagement

RLRF4Rec: Reinforcement Learning from Recsys Feedback for Enhanced Recommendation Reranking

RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation

An End-to-End Deep RL Framework for Task Arrangement in Crowdsourcing Platforms

FedRL: A Reinforcement Learning Federated Recommender System for Efficient Communication Using Reinforcement Selector and Hypernet Generator

Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems

Inference Aided Reinforcement Learning for Incentive Mechanism Design in Crowdsourcing

Model-enhanced Contrastive Reinforcement Learning for Sequential Recommendation

Incentivizing Evaluation via Limited Access to Ground Truth: Peer-Prediction Makes Things Worse

Reward Modeling Requires Automatic Adjustment Based on Data Quality

Eliciting Informative Text Evaluations with Large Language Models

Personalized Language Modeling from Personalized Human Feedback