Abstract:When learning from a batch of logged bandit feedback, the discrepancy between the policy to be learned and the off-policy training data imposes statistical and computational challenges. Unlike classical supervised learning and online learning settings, in batch contextual bandit learning, one only has access to a collection of logged feedback from the actions taken by a historical policy, and expect to learn a policy that takes good actions in possibly unseen contexts. Such a batch learning setting is ubiquitous in online and interactive systems, such as ad platforms and recommendation systems. Existing approaches based on inverse propensity weights, such as Inverse Propensity Scoring (IPS) and Policy Optimizer for Exponential Models (POEM), enjoy unbiasedness but often suffer from large mean squared error. In this work, we introduce a new approach named Maximum Likelihood Inverse Propensity Scoring (MLIPS) for batch learning from logged bandit feedback. Instead of using the given historical policy as the proposal in inverse propensity weights, we estimate a maximum likelihood surrogate policy based on the logged action-context pairs, and then use this surrogate policy as the proposal. We prove that MLIPS is asymptotically unbiased, and moreover, has a smaller nonasymptotic mean squared error than IPS. Such an error reduction phenomenon is somewhat surprising as the estimated surrogate policy is less accurate than the given historical policy. Results on multi-label classification problems and a large- scale ad placement dataset demonstrate the empirical effectiveness of MLIPS. Furthermore, the proposed surrogate policy technique is complementary to existing error reduction techniques, and when combined, is able to consistently boost the performance of several widely used approaches.

MGPolicy: Meta Graph Enhanced Off-policy Learning for Recommendations

MGPolicy

Meta Concept Recommendation Based on Knowledge Graph

Beyond Reward: Offline Preference-guided Policy Optimization

Off-policy Learning over Heterogeneous Information for Recommendation

Off-policy Learning in Two-stage Recommender Systems.

Off-Policy Recommendation System Without Exploration

Reinforcing Long-Term Performance in Recommender Systems with User-Oriented Exploration Policy

Offline Adaptive Policy Leaning in Real-World Sequential Recommendation Systems

UOEP: User-Oriented Exploration Policy for Enhancing Long-Term User Experiences in Recommender Systems

Long Short-Term Temporal Meta-learning in Online Recommendation

Off-Policy Evaluation and Learning from Logged Bandit Feedback: Error Reduction via Surrogate Policy.

CIPPO: Contrastive Imitation Proximal Policy Optimization for Recommendation Based on Reinforcement Learning

An Off-Policy Reinforcement Learning Algorithm Customized for Multi-Task Fusion in Large-Scale Recommender Systems

Model-Based Reinforcement Learning with Adversarial Training for Online Recommendation

M 3 Rec: A Context-aware Offline Meta-level Model-based Reinforcement Learning Approach for Cold-Start Recommendation

Plug-and-Play Model-Agnostic Counterfactual Policy Synthesis for Deep Reinforcement Learning based Recommendation

Pseudo Dyna-Q

Local Policy Improvement for Recommender Systems

Offline Deep Reinforcement Learning Two-stage Optimization Framework Applied to Recommendation Systems

MetaGA: Metalearning with Graph-Attention for Improved Long-Tail Item Recommendation