Abstract:This paper investigates the convergence of learning dynamics in Stackelberg games. In the class of games we consider, there is a hierarchical game being played between a leader and a follower with continuous action spaces. We establish a number of connections between the Nash and Stackelberg equilibrium concepts and characterize conditions under which attracting critical points of simultaneous gradient descent are Stackelberg equilibria in zero-sum games. Moreover, we show that the only stable critical points of the Stackelberg gradient dynamics are Stackelberg equilibria in zero-sum games. Using this insight, we develop a gradient-based update for the leader while the follower employs a best response strategy for which each stable critical point is guaranteed to be a Stackelberg equilibrium in zero-sum games. As a result, the learning rule provably converges to a Stackelberg equilibria given an initialization in the region of attraction of a stable critical point. We then consider a follower employing a gradient-play update rule instead of a best response strategy and propose a two-timescale algorithm with similar asymptotic convergence guarantees. For this algorithm, we also provide finite-time high probability bounds for local convergence to a neighborhood of a stable Stackelberg equilibrium in general-sum games. Finally, we present extensive numerical results that validate our theory, provide insights into the optimization landscape of generative adversarial networks, and demonstrate that the learning dynamics we propose can effectively train generative adversarial networks.

Convergence of Policy Gradient Methods for Nash Equilibria in General-sum Stochastic Games

A Policy-Gradient Approach to Solving Imperfect-Information Games with Iterate Convergence

A Payoff-Based Policy Gradient Method in Stochastic Games with Long-Run Average Payoffs

Gradient play in stochastic games: stationary points, convergence, and sample complexity

Convergence Rate of Payoff-based Generalized Nash Equilibrium Learning

Geometric Convergence of Gradient Play Algorithms for Distributed Nash Equilibrium Seeking

Stability of the Nash Equilibrium under Gradient Ascent Learning Algorithms in Two-Agent Two-Action Games

Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization.

On Gradient-Based Learning in Continuous Games

Convex-Concave Zero-sum Markov Stackelberg Games

Effect of small dysprosium additions on the coercivity of melt-spun Nd-Fe-C alloys

Global Convergence of Policy Gradient Methods in Reinforcement Learning, Games and Control

Global Nash Equilibrium in Non-convex Multi-player Game: Theory and Algorithms

Almost Sure Convergence of Networked Policy Gradient over Time-Varying Networks in Markov Potential Games

Learning in games with continuous action sets and unknown payoff functions

A Policy Iteration Algorithm for N-player General-Sum Linear Quadratic Dynamic Games

Towards convergence to Nash equilibria in two-team zero-sum games

Convergence of Learning Dynamics in Stackelberg Games

Learning generalized Nash equilibria in monotone games: A hybrid adaptive extremum seeking control approach

Convergence and Price of Anarchy Guarantees of the Softmax Policy Gradient in Markov Potential Games

Provable Policy Gradient Methods for Average-Reward Markov Potential Games