Abstract:Recommender systems play a crucial role in our daily lives. Feed streaming mechanism has been widely used in the recommender system, especially on the mobile Apps. The feed streaming setting provides users the interactive manner of recommendation in never-ending feeds. In such an interactive manner, a good recommender system should pay more attention to user stickiness, which is far beyond classical instant metrics, and typically measured by {\bf long-term user engagement}. Directly optimizing the long-term user engagement is a non-trivial problem, as the learning target is usually not available for conventional supervised learning methods. Though reinforcement learning~(RL) naturally fits the problem of maximizing the long term rewards, applying RL to optimize long-term user engagement is still facing challenges: user behaviors are versatile and difficult to model, which typically consists of both instant feedback~(e.g. clicks, ordering) and delayed feedback~(e.g. dwell time, revisit); in addition, performing effective off-policy learning is still immature, especially when combining bootstrapping and function approximation. To address these issues, in this work, we introduce a reinforcement learning framework --- FeedRec to optimize the long-term user engagement. FeedRec includes two components: 1)~a Q-Network which designed in hierarchical LSTM takes charge of modeling complex user behaviors, and 2)~an S-Network, which simulates the environment, assists the Q-Network and voids the instability of convergence in policy learning. Extensive experiments on synthetic data and a real-world large scale data show that FeedRec effectively optimizes the long-term user engagement and outperforms state-of-the-arts.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to optimize users' long - term engagement in the recommendation system. Traditional recommendation systems mainly focus on immediate metrics (such as click - through rate, conversion rate) while ignoring users' long - term behavior and satisfaction, which is especially important in the recommendation scenarios presented in the form of information flow. Users' interactions in the information flow are continuous, so a good recommendation system should pay more attention to users' stickiness, that is, long - term engagement. However, directly optimizing long - term engagement is a non - trivial problem because the learning objective is usually unavailable for traditional supervised learning methods. Although reinforcement learning (RL) is naturally suitable for maximizing long - term rewards, applying RL in the recommendation system to optimize long - term user engagement still faces challenges: 1. **Complex user behavior modeling**: Users' behavior includes both immediate feedback (such as click) and delayed feedback (such as dwell time, revisit). These behaviors are diverse and difficult to model. 2. **Challenges in off - policy learning**: Effective policy learning in an off - line environment is still an immature problem, especially when combining bootstrapping techniques and function approximation, it is prone to instability and divergence problems. To solve these problems, the paper proposes a reinforcement - learning - based framework - FeedRec for optimizing long - term user engagement in the recommendation system. FeedRec includes two main components: 1. **Q - Network**: A hierarchical LSTM structure is designed to model complex user behavior. 2. **S - Network**: Simulate the environment to assist the training of the Q - Network and avoid instability in policy learning. Through extensive experiments on synthetic datasets and large - scale real - world datasets, the paper shows that FeedRec can effectively optimize long - term user engagement and is superior to the existing state - of - the - art methods.

Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems

PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagement

Reinforcing User Retention in a Billion Scale Short Video Recommender System

AdaRec: Adaptive Sequential Recommendation for Reinforcing Long-term User Engagement

ResAct: Reinforcing Long-term Engagement in Sequential Recommendation with Residual Actor

Model-enhanced Contrastive Reinforcement Learning for Sequential Recommendation

A stable deep reinforcement learning framework for recommendation

Sim2Rec: A Simulator-based Decision-making Approach to Optimize Real-World Long-term User Engagement in Sequential Recommender Systems

Optimizing Audio Recommendations for the Long-Term: A Reinforcement Learning Perspective

User Retention-oriented Recommendation with Decision Transformer.

Reinforcing Long-Term Performance in Recommender Systems with User-Oriented Exploration Policy

A Deep Reinforcement Learning Based Long-Term Recommender System

Multi-Task Fusion via Reinforcement Learning for Long-Term User Satisfaction in Recommender Systems

Session-based Interactive Recommendation Via Deep Reinforcement Learning

Reward Shaping for User Satisfaction in a REINFORCE Recommender

RLRF4Rec: Reinforcement Learning from Recsys Feedback for Enhanced Recommendation Reranking

Robust Reinforcement Learning Objectives for Sequential Recommender Systems

Offline Deep Reinforcement Learning Two-stage Optimization Framework Applied to Recommendation Systems

An Off-Policy Reinforcement Learning Algorithm Customized for Multi-Task Fusion in Large-Scale Recommender Systems

Adaptive User Modeling with Long and Short-Term Preferences for Personalized Recommendation.

Optimizing Long-term Value for Auction-Based Recommender Systems via On-Policy Reinforcement Learning