Abstract:This paper investigates the convergence of learning dynamics in Stackelberg games. In the class of games we consider, there is a hierarchical game being played between a leader and a follower with continuous action spaces. We establish a number of connections between the Nash and Stackelberg equilibrium concepts and characterize conditions under which attracting critical points of simultaneous gradient descent are Stackelberg equilibria in zero-sum games. Moreover, we show that the only stable critical points of the Stackelberg gradient dynamics are Stackelberg equilibria in zero-sum games. Using this insight, we develop a gradient-based update for the leader while the follower employs a best response strategy for which each stable critical point is guaranteed to be a Stackelberg equilibrium in zero-sum games. As a result, the learning rule provably converges to a Stackelberg equilibria given an initialization in the region of attraction of a stable critical point. We then consider a follower employing a gradient-play update rule instead of a best response strategy and propose a two-timescale algorithm with similar asymptotic convergence guarantees. For this algorithm, we also provide finite-time high probability bounds for local convergence to a neighborhood of a stable Stackelberg equilibrium in general-sum games. Finally, we present extensive numerical results that validate our theory, provide insights into the optimization landscape of generative adversarial networks, and demonstrate that the learning dynamics we propose can effectively train generative adversarial networks.

On the Global Convergence of Stochastic Fictitious Play in Stochastic Games with Turn-based Controllers

Smooth Fictitious Play in Stochastic Games with Perturbed Payoffs and Unknown Transitions

Convergence of Heterogeneous Learning Dynamics in Zero-sum Stochastic Games

Fictitious Play in Markov Games with Single Controller

Last-Iterate Convergence of Payoff-Based Independent Learning in Zero-Sum Stochastic Games

Convergence of Learning Dynamics in Stackelberg Games

Convergence of deep fictitious play for stochastic differential games

Convergence and Stability of Coupled Belief-Strategy Learning Dynamics in Continuous Games

Deep Fictitious Play for Stochastic Differential Games

Playing Against Fair Adversaries in Stochastic Games with Total Rewards

Learning the Optimal Control for Evolving Systems with Converging Dynamics

Finite-horizon Approximations and Episodic Equilibrium for Stochastic Games

Independent Learning in Stochastic Games

Convergence of Decentralized Actor-Critic Algorithm in General-sum Markov Games

The equivalence of dynamic and strategic stability under regularized learning in games

On the Convergence of No-Regret Learning Dynamics in Time-Varying Games

Stochastic Shortest Path Games and Q-Learning

Stochastic Games with Minimally Bounded Action Costs

A zero-sum hybrid stochastic differential game with switching controls

On the Convergence of Fictitious Play: A Decomposition Approach

On Convergence Rates of Robust Adaptive Game Theoretic Learning Algorithms