Abstract:Learning in general-sum games often yields collectively sub-optimal results. Addressing this, opponent shaping (OS) methods actively guide the learning processes of other agents, empirically leading to improved individual and group performances in many settings. Early OS methods use higher-order derivatives to shape the learning of co-players, making them unsuitable for shaping multiple learning steps. Follow-up work, Model-free Opponent Shaping (M-FOS), addresses these by reframing the OS problem as a meta-game. In contrast to early OS methods, there is little theoretical understanding of the M-FOS framework. Providing theoretical guarantees for M-FOS is hard because A) there is little literature on theoretical sample complexity bounds for meta-reinforcement learning B) M-FOS operates in continuous state and action spaces, so theoretical analysis is challenging. In this work, we present R-FOS, a tabular version of M-FOS that is more suitable for theoretical analysis. R-FOS discretises the continuous meta-game MDP into a tabular MDP. Within this discretised MDP, we adapt the $R_{max}$ algorithm, most prominently used to derive PAC-bounds for MDPs, as the meta-learner in the R-FOS algorithm. We derive a sample complexity bound that is exponential in the cardinality of the inner state and action space and the number of agents. Our bound guarantees that, with high probability, the final policy learned by an R-FOS agent is close to the optimal policy, apart from a constant factor. Finally, we investigate how R-FOS's sample complexity scales in the size of state-action space. Our theoretical results on scaling are supported empirically in the Matching Pennies environment.

What problem does this paper attempt to address?

The main focus of this paper is the sample complexity analysis of the Opponent Shaping (OS) method in multi-agent systems. Specifically, the paper studies a method called Model-free Opponent Shaping (M-FOS), which improves individual and group performance by guiding the learning process of other agents within a meta-game framework. Early opponent shaping methods used higher-order derivatives to influence other players' learning processes, leading to algorithm instability and difficulty in scaling to multiple learning steps. The M-FOS method addresses these issues by reformulating the opponent shaping problem as a meta-game, where each meta-step represents a set of embedded "inner games." However, theoretical understanding of M-FOS is very limited, especially in terms of sample complexity. To provide theoretical guarantees for M-FOS, the authors propose a tabular version algorithm called R-FOS. R-FOS discretizes the continuous state and action spaces of the meta-Markov Decision Process (meta-MDP), making theoretical analysis possible. In the discretized meta-MDP, the authors apply an algorithm called 𝑅𝑚𝑎𝑥 as the meta-learner, which is a model-based reinforcement learning algorithm commonly used to derive sample complexity bounds for Markov Decision Processes. The authors derive an exponential sample complexity bound, which is exponential in the cardinality of the internal state and action spaces and the number of agents. This bound ensures that the final policy learned by R-FOS is within a constant factor of the optimal policy. Additionally, the authors study how the sample complexity of R-FOS changes with the size of the state-action space and validate the theoretical results through experiments. In summary, the problem this paper attempts to solve is to provide a theoretical sample complexity analysis of the M-FOS algorithm, particularly evaluating its efficiency in opponent shaping in multi-agent environments.

Analysing the Sample Complexity of Opponent Shaping

Scaling Opponent Shaping to High Dimensional Games

Leading the Pack: N-player Opponent Shaping

Preference-based opponent shaping in differentiable games

RM-FSP: Regret Minimization Optimizes Neural Fictitious Self-Play

Self-Supervised Online Reward Shaping in Sparse-Reward Environments

Learning to Shape Rewards Using a Game of Two Partners

Best Response Shaping

Modeling opponent learning in multiagent repeated games

Improving Sample Efficiency of Model-Free Algorithms for Zero-Sum Markov Games

Opponent Learning Awareness and Modelling in Multi-Objective Normal Form Games

Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts

School of hard knocks: Curriculum analysis for Pommerman with a fixed computational budget

When Can We Learn General-Sum Markov Games with a Large Number of Players Sample-Efficiently?

Online meta-learning by parallel algorithm competition

Finite-Sample Analysis of the Monte Carlo Exploring Starts Algorithm for Reinforcement Learning

Breaking the sample complexity barrier to regret-optimal model-free reinforcement learning

Posterior Sampling for Competitive RL: Function Approximation and Partial Observation

Instance-Dependent Near-Optimal Policy Identification in Linear MDPs via Online Experiment Design

Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learning

A Sharp Analysis of Model-based Reinforcement Learning with Self-Play