Abstract: Long-term user engagement (LTE) optimization in sequential recommender systems (SRS) is shown to be suited by reinforcement learning (RL) which finds a policy to maximize long-term rewards. Meanwhile, RL has its shortcomings, particularly requiring a large number of online samples for exploration, which is risky in real-world applications. One of the appealing ways to avoid the risk is to build a simulator and learn the optimal recommendation policy in the simulator. In LTE optimization, the simulator is to simulate multiple users' daily feedback for given recommendations. However, building a user simulator with no reality-gap, i.e., can predict user's feedback exactly, is unrealistic because the users' reaction patterns are complex and historical logs for each user are limited, which might mislead the simulator-based recommendation policy. In this paper, we present a practical simulator-based recommender policy training approach, Simulation-to-Recommendation (Sim2Rec) to handle the reality-gap problem for LTE optimization. Specifically, Sim2Rec introduces a simulator set to generate various possibilities of user behavior patterns, then trains an environment-parameter extractor to recognize users' behavior patterns in the simulators. Finally, a context-aware policy is trained to make the optimal decisions on all of the variants of the users based on the inferred environment-parameters. The policy is transferable to unseen environments (e.g., the real world) directly as it has learned to recognize all various user behavior patterns and to make the correct decisions based on the inferred environment-parameters. Experiments are conducted in synthetic environments and a real-world large-scale ride-hailing platform, DidiChuxing. The results show that Sim2Rec achieves significant performance improvement, and produces robust recommendations in unseen environments.

SARDINE: A Simulator for Automated Recommendation in Dynamic and Interactive Environments

Toward Simulating Environments in Reinforcement Learning Based Recommendations.

KuaiSim: A Comprehensive Simulator for Recommender Systems

Towards Simulation-Based Evaluation of Recommender Systems with Carousel Interfaces

Offline Evaluation of Reward-Optimizing Recommender Systems: The Case of Simulation

RecAgent: A Novel Simulation Paradigm for Recommender Systems

Sim-to-Real Interactive Recommendation via Off-Dynamics Reinforcement Learning

Sim2Rec: A Simulator-based Decision-making Approach to Optimize Real-World Long-term User Engagement in Sequential Recommender Systems

RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems

A LLM-based Controllable, Scalable, Human-Involved User Simulator Framework for Conversational Recommender Systems

On Generative Agents in Recommendation

What Else Would I Like? A User Simulator using Alternatives for Improved Evaluation of Fashion Conversational Recommendation Systems

Algorithmic Drift: A Simulation Framework to Study the Effects of Recommender Systems on User Preferences

Synthetic Data and Simulators for Recommendation Systems: Current State and Future Directions

LLM-Powered User Simulator for Recommender System

Exploiting Structural and Temporal Influence for Dynamic Social-Aware Recommendation

UserSim: User Simulation via Supervised Generative Adversarial Network

MINDSim: User Simulator for News Recommenders

Minimizing Live Experiments in Recommender Systems: User Simulation to Evaluate Preference Elicitation Policies

UserSimCRS: A User Simulation Toolkit for Evaluating Conversational Recommender Systems

User Behavior Simulation with Large Language Model-based Agents for Recommender Systems