Abstract: Long-term user engagement (LTE) optimization in sequential recommender systems (SRS) is shown to be suited by reinforcement learning (RL) which finds a policy to maximize long-term rewards. Meanwhile, RL has its shortcomings, particularly requiring a large number of online samples for exploration, which is risky in real-world applications. One of the appealing ways to avoid the risk is to build a simulator and learn the optimal recommendation policy in the simulator. In LTE optimization, the simulator is to simulate multiple users' daily feedback for given recommendations. However, building a user simulator with no reality-gap, i.e., can predict user's feedback exactly, is unrealistic because the users' reaction patterns are complex and historical logs for each user are limited, which might mislead the simulator-based recommendation policy. In this paper, we present a practical simulator-based recommender policy training approach, Simulation-to-Recommendation (Sim2Rec) to handle the reality-gap problem for LTE optimization. Specifically, Sim2Rec introduces a simulator set to generate various possibilities of user behavior patterns, then trains an environment-parameter extractor to recognize users' behavior patterns in the simulators. Finally, a context-aware policy is trained to make the optimal decisions on all of the variants of the users based on the inferred environment-parameters. The policy is transferable to unseen environments (e.g., the real world) directly as it has learned to recognize all various user behavior patterns and to make the correct decisions based on the inferred environment-parameters. Experiments are conducted in synthetic environments and a real-world large-scale ride-hailing platform, DidiChuxing. The results show that Sim2Rec achieves significant performance improvement, and produces robust recommendations in unseen environments.

Building Personalized Simulator for Interactive Search.

Learning through Dialogue Interactions by Asking Questions

User Behavior Simulation for Search Result Re-ranking

Interactive Search Based on Deep Reinforcement Learning

USimAgent: Large Language Models for Simulating Search Users

KuaiSim: A Comprehensive Simulator for Recommender Systems

Toward Simulating Environments in Reinforcement Learning Based Recommendations.

Sim2Rec: A Simulator-based Decision-making Approach to Optimize Real-World Long-term User Engagement in Sequential Recommender Systems

Joint Learning of Interactive Spoken Content Retrieval and Trainable User Simulator.

RLPS: A Reinforcement Learning–Based Framework for Personalized Search

AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systems

On Generative Agents in Recommendation

Imitate TheWorld: A Search Engine Simulation Platform

A User Simulator for Task-Completion Dialogues

Deep Reinforcement Learning for Personalized Search Story Recommendation

A Deep Reinforcement Learning Approach for Interactive Search with Sentence-level Feedback

Sim-to-Real Interactive Recommendation via Off-Dynamics Reinforcement Learning

An In-depth Investigation of User Response Simulation for Conversational Search.

BASES: Large-scale Web Search User Simulation with Large Language Model based Agents

An Iterative Implicit Feedback Approach To Personalized Search