Abstract:Computing Nash equilibrium policies is a central problem in multi-agent reinforcement learning that has received extensive attention both in theory and in practice. However, provable guarantees have been thus far either limited to fully competitive or cooperative scenarios or impose strong assumptions that are difficult to meet in most practical applications. In this work, we depart from those prior results by investigating infinite-horizon \emph{adversarial team Markov games}, a natural and well-motivated class of games in which a team of identically-interested players -- in the absence of any explicit coordination or communication -- is competing against an adversarial player. This setting allows for a unifying treatment of zero-sum Markov games and Markov potential games, and serves as a step to model more realistic strategic interactions that feature both competing and cooperative interests. Our main contribution is the first algorithm for computing stationary $\epsilon$-approximate Nash equilibria in adversarial team Markov games with computational complexity that is polynomial in all the natural parameters of the game, as well as $1/\epsilon$. The proposed algorithm is particularly natural and practical, and it is based on performing independent policy gradient steps for each player in the team, in tandem with best responses from the side of the adversary; in turn, the policy for the adversary is then obtained by solving a carefully constructed linear program. Our analysis leverages non-standard techniques to establish the KKT optimality conditions for a nonlinear program with nonconvex constraints, thereby leading to a natural interpretation of the induced Lagrange multipliers. Along the way, we significantly extend an important characterization of optimal policies in adversarial (normal-form) team games due to Von Stengel and Koller (GEB `97).

Nash Equilibria and Pitfalls of Adversarial Training in Adversarial Robustness Games

Adversarial Training Should Be Cast as a Non-Zero-Sum Game

Adversaries in Online Learning Revisited: with applications in Robust Optimization and Adversarial training

Adversaries With Incentives: A Strategic Alternative to Adversarial Robustness

Precise Tradeoffs in Adversarial Training for Linear Regression

Adversarial Training and Robustness for Multiple Perturbations

Robust Adversarial Reinforcement Learning via Bounded Rationality Curricula

Fundamental Tradeoffs in Distributionally Adversarial Training

Adversarial Training with Anti-adversaries

A Closer Look at the Adversarial Robustness of Deep Equilibrium Models

A Game Theoretic Analysis of Additive Adversarial Attacks and Defenses

Achieve Optimal Adversarial Accuracy for Adversarial Deep Learning using Stackelberg Game

Revisiting the Adversarial Robustness-Accuracy Tradeoff in Robot Learning

How robust accuracy suffers from certified training with convex relaxations

The Pros and Cons of Adversarial Robustness

Efficiently Computing Nash Equilibria in Adversarial Team Markov Games

To be Robust or to be Fair: Towards Fairness in Adversarial Training

On the Robustness of Adversarial Training Against Uncertainty Attacks

On Convergence Rates of Robust Adaptive Game Theoretic Learning Algorithms

A Robust Characterization of Nash Equilibrium

Local Competition and Uncertainty for Adversarial Robustness in Deep Learning