Abstract:For Internet applications like sponsored search, cautions need to be taken when using machine learning to optimize their mechanisms (e.g., auction) since self-interested agents in these applications may change their behaviors (and thus the data distribution) in response to the mechanisms. To tackle this problem, a framework called game-theoretic machine learning (GTML) was recently proposed, which first learns a Markov behavior model to characterize agents' behaviors, and then learns the optimal mechanism by simulating agents' behavior changes in response to the mechanism. While GTML has demonstrated practical success, its generalization analysis is challenging because the behavior data are non-i.i.d. and dependent on the mechanism. To address this challenge, first, we decompose the generalization error for GTML into the behavior learning error and the mechanism learning error; second, for the behavior learning error, we obtain novel non-asymptotic error bounds for both parametric and non-parametric behavior learning methods; third, for the mechanism learning error, we derive a uniform convergence bound based on a new concept called nested covering number of the mechanism space and the generalization analysis techniques developed for mixing sequences. To the best of our knowledge, this is the first work on the generalization analysis of GTML, and we believe it has general implications to the theoretical analysis of other complicated machine learning problems.

A Game-Theoretic Perspective of Generalization in Reinforcement Learning

Understanding What Affects the Generalization Gap in Visual Reinforcement Learning: Theory and Empirical Evidence

Learning to Play General-Sum Games against Multiple Boundedly Rational Agents

Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentation

A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning

Discovering General Reinforcement Learning Algorithms with Adversarial Environment Design

Dynamics Generalization via Information Bottleneck in Deep Reinforcement Learning

Towards Understanding How to Reduce Generalization Gap in Visual Reinforcement Learning.

Theoretical Analysis of Meta Reinforcement Learning: Generalization Bounds and Convergence Guarantees

Learning Generalizable Risk-Sensitive Policies to Coordinate in Decentralized Multi-Agent General-Sum Games

Grounded Reinforcement Learning: Learning to Win the Game under Human Commands

Curriculum in Gradient-Based Meta-Reinforcement Learning

Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training

Generalized Multi-Agent Competitive Reinforcement Learning with Differential Augmentation

Active Reinforcement Learning over MDPs

Improving Generalization in Reinforcement Learning Training Regimes for Social Robot Navigation

Generalization and Regularization in DQN

Robust Reinforcement Learning through Efficient Adversarial Herding

Automatic Data Augmentation for Generalization in Reinforcement Learning

Towards Generalized Inverse Reinforcement Learning

Generalization Analysis for Game-Theoretic Machine Learning