Abstract:A recent method for solving zero-sum partially observable stochastic games (zs-POSGs) embeds the original game into a new one called the occupancy Markov game. This reformulation allows applying Bellman's principle of optimality to solve zs-POSGs. However, improving a current solution requires solving a linear program with exponentially many potential constraints, which significantly restricts the scalability of this approach. This paper exploits the optimal value function's novel uniform continuity properties to overcome this limitation. We first construct a new operator that is computationally more efficient than the state-of-the-art update rules without compromising optimality. In particular, improving a current solution now involves a linear program with an exponential drop in constraints. We then also show that point-based value iteration algorithms utilizing our findings improve the scalability of existing methods while maintaining guarantees in various domains.

What problem does this paper attempt to address?

This paper attempts to solve the scalability problem encountered when finding optimal strategies in zero - sum partially observable stochastic games (zs - POSGs). Specifically, existing methods apply the Bellman optimality principle by embedding the original game into a new fully observable game called the occupancy Markov game (OMG). However, improving the current solution requires solving a linear programming problem with exponentially potential constraints, which greatly limits the scalability of this method. To solve this problem, the paper takes advantage of the novel uniform continuity property of the optimal value function. The author first constructs a new operator that improves computational efficiency without sacrificing optimality, especially by reducing the number of constraints to improve the current solution. In addition, the paper also shows that using these newly discovered point - based value iteration algorithms can improve the scalability of existing methods while maintaining guarantees in various fields. ### Specific problem description 1. **Limitations of existing methods**: - **Curse of dimensionality**: The states of OMG (i.e., occupancy states) exist in a continuous space whose dimension grows exponentially over time. - **Inefficiency of the update operator**: State - of - the - art update operators use linear programming with exponentially potential constraints, resulting in limited scalability in zero - sum settings. 2. **Main contributions of the paper**: - **Uniform continuity property**: It is proved that the optimal value function corresponding to OMG is the maximum of piecewise - linear and concave functions in the occupancy state. - **Efficient update operator**: A two - part update operator is constructed: 1. A linear programming with exponentially reduced constraints, enabling any sub - game to be solved efficiently. 2. A polynomial - time update rule that uses the solutions of sub - games to improve the current value function. - **Point - based value iteration algorithm**: It is shown how to use this update operator in the point - based value iteration algorithm, and its effectiveness is verified by experimental results. ### Mathematical formula representation To better understand the method of the paper, the following are the Markdown - format representations of several key formulas: - **Bellman equation**: \[ \upsilon^*_\tau(s_\tau) = \max_{a^\star_\tau \in A^\star_\tau} \min_{a^\circ_\tau \in A^\circ_\tau} R(s_\tau, a_\tau) + \gamma \upsilon^*_{\tau+1}(T(s_\tau, a_\tau)) \] where \( s_\tau \) is the occupancy state, \( a_\tau \) is the joint decision rule, \( R(s_\tau, a_\tau) \) is the reward function, and \( T(s_\tau, a_\tau) \) is the transition rule. - **Expression of the optimal value function**: \[ \upsilon^*_\tau(s_\tau) = \max_{V_\tau \in V^*_\tau} \sum_{o^\circ \in O^\circ} s_m^\circ(o^\circ) \min_{\alpha_\tau \in V_\tau} \alpha_\tau(s_c, o^\circ) \] where \( V^*_\tau \) is the set of linear functions on the conditional occupancy state. - **Linear programming problem**: \[ \begin{aligned} & \text{Maximize} & & \sum_{o^\circ \in O^\circ} \alpha_\theta(o^\circ) \\ & \text{Subject to} & & \alpha_\theta(o^\circ) \le

$ε$-Optimally Solving Zero-Sum POSGs

Advances in finding ideal play on poset games

Linear-quadratic zero-sum mean-field type games: Optimality conditions and policy optimization

Convex-Concave Zero-sum Markov Stackelberg Games

Zero-Sum Stochastic Stackelberg Games

Strategy investments in zero-sum games

Bridging the Gap between Partially Observable Stochastic Games and Sparse POMDP Methods

Heuristic Search Value Iteration for One-Sided Partially Observable Stochastic Games

A set optimization approach to zero-sum matrix games with multi-dimensional payoffs

Universal Complexity Bounds Based on Value Iteration for Stochastic Mean Payoff Games and Entropy Games

Beyond Minimax Optimality: A Subgame Perfect Gradient Method

Near-Optimal Last-iterate Convergence of Policy Optimization in Zero-sum Polymatrix Markov Games

A Self-Play Posterior Sampling Algorithm for Zero-Sum Markov Games

Sparsified Linear Programming for Zero-Sum Equilibrium Finding

Towards Solving 2-TBSG Efficiently

Anytime PSRO for Two-Player Zero-Sum Games

Stochastic Games with Minimally Bounded Action Costs

The Zero Regrets Algorithm: Optimizing over Pure Nash Equilibria via Integer Programming

Stackelberg-Pareto Synthesis with Quantitative Reachability Objectives

Universally Balanced Combinatorial Optimization Games

Optimal Doubling Thresholds in Backgammon-like Stochastic Games