Abstract: Long-term user engagement (LTE) optimization in sequential recommender systems (SRS) is shown to be suited by reinforcement learning (RL) which finds a policy to maximize long-term rewards. Meanwhile, RL has its shortcomings, particularly requiring a large number of online samples for exploration, which is risky in real-world applications. One of the appealing ways to avoid the risk is to build a simulator and learn the optimal recommendation policy in the simulator. In LTE optimization, the simulator is to simulate multiple users' daily feedback for given recommendations. However, building a user simulator with no reality-gap, i.e., can predict user's feedback exactly, is unrealistic because the users' reaction patterns are complex and historical logs for each user are limited, which might mislead the simulator-based recommendation policy. In this paper, we present a practical simulator-based recommender policy training approach, Simulation-to-Recommendation (Sim2Rec) to handle the reality-gap problem for LTE optimization. Specifically, Sim2Rec introduces a simulator set to generate various possibilities of user behavior patterns, then trains an environment-parameter extractor to recognize users' behavior patterns in the simulators. Finally, a context-aware policy is trained to make the optimal decisions on all of the variants of the users based on the inferred environment-parameters. The policy is transferable to unseen environments (e.g., the real world) directly as it has learned to recognize all various user behavior patterns and to make the correct decisions based on the inferred environment-parameters. Experiments are conducted in synthetic environments and a real-world large-scale ride-hailing platform, DidiChuxing. The results show that Sim2Rec achieves significant performance improvement, and produces robust recommendations in unseen environments.

Combining Open-box Simulation and Importance Sampling for Tuning Large-Scale Recommenders

Adaptive Mixture Importance Sampling for Automated Ads Auction Tuning

KuaiSim: A Comprehensive Simulator for Recommender Systems

RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems

Towards Reliable Item Sampling for Recommendation Evaluation

AutoSAM: Towards Automatic Sampling of User Behaviors for Sequential Recommender Systems

Learning Recommenders for Implicit Feedback with Importance Resampling

SARDINE: A Simulator for Automated Recommendation in Dynamic and Interactive Environments

CoSam: An Efficient Collaborative Adaptive Sampler for Recommendation

MultiSlot ReRanker: A Generic Model-based Re-Ranking Framework in Recommendation Systems

Toward Simulating Environments in Reinforcement Learning Based Recommendations.

Sim2Rec: A Simulator-based Decision-making Approach to Optimize Real-World Long-term User Engagement in Sequential Recommender Systems

Jointly Learning to Recommend and Advertise

Minimizing Live Experiments in Recommender Systems: User Simulation to Evaluate Preference Elicitation Policies

Reactive Sample Size for Heuristic Search in Simulation-based Optimization

Cache-Augmented Inbatch Importance Resampling for Training Recommender Retriever

Multi-fidelity Constrained Optimization for Stochastic Black Box Simulators

Entire Space Learning Framework: Unbias Conversion Rate Prediction in Full Stages of Recommender System

Cache-Augmented Inbatch Importance Resampling for Training Recommender Retriever

Training Recommenders over Large Item Corpus with Importance Sampling

Algorithmic Drift: A Simulation Framework to Study the Effects of Recommender Systems on User Preferences