Abstract:We study a subclass of $n$-player stochastic games, namely, stochastic games with independent chains and unknown transition matrices. In this class of games, players control their own internal Markov chains whose transitions do not depend on the states/actions of other players. However, players' decisions are coupled through their payoff functions. We assume players can receive only realizations of their payoffs, and that the players can not observe the states and actions of other players, nor do they know the transition probability matrices of their own Markov chain. Relying on a compact dual formulation of the game based on occupancy measures and the technique of confidence set to maintain high-probability estimates of the unknown transition matrices, we propose a fully decentralized mirror descent algorithm to learn an $\epsilon$-NE for this class of games. The proposed algorithm has the desired properties of independence, scalability, and convergence. Specifically, under no assumptions on the reward functions, we show the proposed algorithm converges in polynomial time in a weaker distance (namely, the averaged Nikaido-Isoda gap) to the set of $\epsilon$-NE policies with arbitrarily high probability. Moreover, assuming the existence of a variationally stable Nash equilibrium policy, we show that the proposed algorithm converges asymptotically to the stable $\epsilon$-NE policy with arbitrarily high probability. In addition to Markov potential games and linear-quadratic stochastic games, this work provides another subclass of $n$-player stochastic games that, under some mild assumptions, admit polynomial-time learning algorithms for finding their stationary $\epsilon$-NE policies.

What problem does this paper attempt to address?

This paper attempts to solve the problem of how players can independently and scalably learn Nash Equilibrium Policies in n - person stochastic games with independent chains and unknown transition matrices. Specifically, the paper focuses on the following challenges: 1. **Independence**: Each player can only make decisions based on their own state and reward information, and cannot observe the states or actions of other players. 2. **Scalability**: The algorithm needs to be able to handle a large number of players, and the computational complexity should remain within an acceptable range as the number of players increases. 3. **Convergence**: The proposed algorithm should converge to an approximate Nash Equilibrium Policy ($\epsilon$-NE) within a reasonable time, and this convergence should be guaranteed with high confidence in a probabilistic sense. ### Specific Problem Description in the Paper The paper studies a special class of n - person stochastic games, namely stochastic games with independent chains and unknown transition matrices. In such games, each player controls an internal Markov Chain, whose transition probabilities do not depend on the states or actions of other players, but the players' decisions are coupled together through the payoff function. In addition, players can only obtain the realized values of their own payoffs, cannot observe the states or actions of other players, and do not know the transition probability matrix of their own Markov Chain. ### Overview of the Solution To address the above challenges, the paper proposes a fully decentralized mirror descent algorithm. This algorithm is based on the dual formulation of occupancy measures and confidence set techniques to maintain a high - probability estimate of the unknown transition matrix. The main features of this algorithm are as follows: - **Independence**: Each player updates their strategy only based on their own observed data, without coordinating with other players. - **Scalability**: The algorithm converges to an approximate Nash Equilibrium Policy in polynomial time and is suitable for large - scale games. - **Convergence**: Under a weaker distance metric (such as the average Nikaido - Isoda gap), the algorithm converges to the $\epsilon$-NE policy set with arbitrarily high probability. If there is a variationally stable Nash Equilibrium Policy, the algorithm will asymptotically converge to this stable point. ### Application Examples The paper provides two practical application examples to illustrate the importance of this type of stochastic game: 1. **Energy Management in Smart Grids**: Users interact with power companies to learn their respective energy consumption/production levels in order to maximize the total revenue. 2. **Power Allocation in Multi - Agent Wireless Communications**: Senders adjust the transmission power according to the received successful transmission rewards to optimize the communication effect. ### Summary The main contribution of this paper is to propose a new decentralized algorithm that can efficiently and independently learn approximate Nash Equilibrium Policies in n - person stochastic games with independent chains and unknown transition matrices. This provides an effective solution for solving the learning problems in large - scale non - cooperative games.

Scalable and Independent Learning of Nash Equilibrium Policies in $n$-Player Stochastic Games with Unknown Independent Chains

Learning Stationary Nash Equilibrium Policies in [math]-Player Stochastic Games with Independent Chains

Independent Learning in Constrained Markov Potential Games

Learning to Control Unknown Strongly Monotone Games

Convergent Learning Algorithms for Unknown Reward Games

Empirical Policy Optimization for n-Player Markov Games

Reinforcement Nash Equilibrium Solver

Approximate Nash Equilibrium Learning for n-Player Markov Games in Dynamic Pricing

Independent and Decentralized Learning in Markov Potential Games

Provably Fast Convergence of Independent Natural Policy Gradient for Markov Potential Games

Empirical Policy Optimization for <i>n</i>-Player Markov Games

Efficiently Computing Nash Equilibria in Adversarial Team Markov Games

Decentralized and Uncoordinated Learning of Stable Matchings: A Game-Theoretic Approach

Optimistic Policy Gradient in Multi-Player Markov Games with a Single Controller: Convergence Beyond the Minty Property

Independent RL for Cooperative-Competitive Agents: A Mean-Field Perspective

Multiplayer Stackelberg-Nash Game for Nonlinear System via Value Iteration-Based Integral Reinforcement Learning

Deep Fictitious Play for Finding Markovian Nash Equilibrium in Multi-Agent Games

Learning Nash Equilibria in Zero-Sum Markov Games: A Single Time-scale Algorithm Under Weak Reachability

Off-policy Q-learning: Solving Nash Equilibrium of Multi-Player Games with Network-Induced Delay and Unmeasured State.

Breaking the Curse of Multiagents in a Large State Space: RL in Markov Games with Independent Linear Function Approximation

A Neural-Network based Approach for Nash Equilibrium Seeking in Mixed-order Multi-player Games