The Complexity of Simple Stochastic Games

Jonas Dieckelmann

DOI: https://doi.org/10.48550/arXiv.0704.2779

2007-04-21

Abstract:In this paper we survey the computational time complexity of assorted simple stochastic game problems, and we give an overview of the best known algorithms associated with each problem.

Computational Complexity,Computer Science and Game Theory

What problem does this paper attempt to address?

Comparison of Algorithms for Simple Stochastic Games (Full Version)

Jan Kretinsky,Emanuel Ramneantu,Alexander Slivinskiy,Maximilian Weininger

DOI: https://doi.org/10.4204/EPTCS.326.9

2020-08-25

Abstract:Simple stochastic games are turn-based 2.5-player zero-sum graph games with a reachability objective. The problem is to compute the winning probability as well as the optimal strategies of both players. In this paper, we compare the three known classes of algorithms -- value iteration, strategy iteration and quadratic programming -- both theoretically and practically. Further, we suggest several improvements for all algorithms, including the first approach based on quadratic programming that avoids transforming the stochastic game to a stopping one. Our extensive experiments show that these improvements can lead to significant speed-ups. We implemented all algorithms in PRISM-games 3.0, thereby providing the first implementation of quadratic programming for solving simple stochastic games.

Computer Science and Game Theory,Data Structures and Algorithms
Comparison of Algorithms for Simple Stochastic Games

Jan Křetínský,Emanuel Ramneantu,Alexander Slivinskiy,Maximilian Weininger

DOI: https://doi.org/10.4204/EPTCS.326.9

2020-09-23

Abstract:Simple stochastic games are turn-based 2.5-player zero-sum graph games with a reachability objective. The problem is to compute the winning probability as well as the optimal strategies of both players. In this paper, we compare the three known classes of algorithms -- value iteration, strategy iteration and quadratic programming -- both theoretically and practically. Further, we suggest several improvements for all algorithms, including the first approach based on quadratic programming that avoids transforming the stochastic game to a stopping one. Our extensive experiments show that these improvements can lead to significant speed-ups. We implemented all algorithms in PRISM-games 3.0, thereby providing the first implementation of quadratic programming for solving simple stochastic games.

Computer Science and Game Theory,Data Structures and Algorithms
On the computational complexity of solving stochastic mean-payoff games

Vladimir Gurvich,Peter Bro Miltersen

DOI: https://doi.org/10.48550/arXiv.0812.0486

2008-12-02

Computer Science and Game Theory

Abstract:We consider some well-known families of two-player, zero-sum, perfect information games that can be viewed as special cases of Shapley's stochastic games. We show that the following tasks are polynomial time equivalent: - Solving simple stochastic games. - Solving stochastic mean-payoff games with rewards and probabilities given in unary. - Solving stochastic mean-payoff games with rewards and probabilities given in binary.
Exact Algorithms for Solving Stochastic Games

Kristoffer Arnsfelt Hansen,Michal Koucky,Niels Lauritzen,Peter Bro Miltersen,Elias Tsigaridas

DOI: https://doi.org/10.48550/arXiv.1202.3898

2012-02-17

Abstract:Shapley's discounted stochastic games, Everett's recursive games and Gillette's undiscounted stochastic games are classical models of game theory describing two-player zero-sum games of potentially infinite duration. We describe algorithms for exactly solving these games.

Computer Science and Game Theory
Simple Stochastic Stopping Games: A Generator and Benchmark Library

Avi Rudich,Isaac Rudich,Rachel Rue

2024-02-05

Abstract:Simple Stochastic Games (SSGs) were introduced by Anne Condon in 1990, as the simplest version of Stochastic Games for which there is no known polynomial-time algorithm. Condon showed that Stochastic Games are polynomial-time reducible to SSGs, which in turn are polynomial-time reducible to Stopping Games. SSGs are games where all decisions are binary and every move has a random outcome with a known probability distribution. Stopping Games are SSGs that are guaranteed to terminate. There are many algorithms for SSGs, most of which are fast in practice, but they all lack theoretical guarantees for polynomial-time convergence. The pursuit of a polynomial-time algorithm for SSGs is an active area of research. This paper is intended to support such research by making it easier to study the graphical structure of SSGs. Our contributions are: (1) a generating algorithm for Stopping Games, (2) a proof that the algorithm can generate any game, (3) a list of additional polynomial-time reductions that can be made to Stopping Games, (4) an open source generator for generating fully reduced instances of Stopping Games that comes with instructions and is fully documented, (5) a benchmark set of such instances, (6) and an analysis of how two main algorithm types perform on our benchmark set.

Computational Complexity,Computer Science and Game Theory
Optimal Time-Abstract Schedulers for CTMDPs and Markov Games

Markus Rabe,Sven Schewe

DOI: https://doi.org/10.4204/EPTCS.28.10

2010-06-26

Abstract:We study time-bounded reachability in continuous-time Markov decision processes for time-abstract scheduler classes. Such reachability problems play a paramount role in dependability analysis and the modelling of manufacturing and queueing systems. Consequently, their analysis has been studied intensively, and techniques for the approximation of optimal control are well understood. From a mathematical point of view, however, the question of approximation is secondary compared to the fundamental question whether or not optimal control exists. We demonstrate the existence of optimal schedulers for the time-abstract scheduler classes for all CTMDPs. Our proof is constructive: We show how to compute optimal time-abstract strategies with finite memory. It turns out that these optimal schedulers have an amazingly simple structure - they converge to an easy-to-compute memoryless scheduling policy after a finite number of steps. Finally, we show that our argument can easily be lifted to Markov games: We show that both players have a likewise simple optimal strategy in these more general structures.

Formal Languages and Automata Theory
Simplified Game of Life: Algorithms and Complexity

Krishnendu Chatterjee,Rasmus Ibsen-Jensen,Ismaël Jecker,Jakub Svoboda

DOI: https://doi.org/10.4230/LIPIcs.MFCS.2020.22

2020-07-07

Abstract:Game of Life is a simple and elegant model to study dynamical system over networks. The model consists of a graph where every vertex has one of two types, namely, dead or alive. A configuration is a mapping of the vertices to the types. An update rule describes how the type of a vertex is updated given the types of its neighbors. In every round, all vertices are updated synchronously, which leads to a configuration update. While in general, Game of Life allows a broad range of update rules, we focus on two simple families of update rules, namely, underpopulation and overpopulation, that model several interesting dynamics studied in the literature. In both settings, a dead vertex requires at least a desired number of live neighbors to become alive. For underpopulation (resp., overpopulation), a live vertex requires at least (resp. at most) a desired number of live neighbors to remain alive. We study the basic computation problems, e.g., configuration reachability, for these two families of rules. For underpopulation rules, we show that these problems can be solved in polynomial time, whereas for overpopulation rules they are PSPACE-complete.

Cellular Automata and Lattice Gases
On the complexity of computing Markov perfect equilibrium in general-sum stochastic games

Xiaotie Deng,Ningyuan Li,David Mguni,Jun Wang,Yaodong Yang

DOI: https://doi.org/10.1093/nsr/nwac256

IF: 20.6

2022-11-22

National Science Review

Abstract:ABSTRACT Similar to the role of Markov decision processes in reinforcement learning, Markov games (also called stochastic games) lay down the foundation for the study of multi-agent reinforcement learning and sequential agent interactions. We introduce approximate Markov perfect equilibrium as a solution to the computational problem of finite-state stochastic games repeated in the infinite horizon and prove its PPAD-completeness. This solution concept preserves the Markov perfect property and opens up the possibility for the success of multi-agent reinforcement learning algorithms on static two-player games to be extended to multi-agent dynamic games, expanding the reign of the PPAD-complete class.

multidisciplinary sciences
Universal Complexity Bounds Based on Value Iteration for Stochastic Mean Payoff Games and Entropy Games

Xavier Allamigeon,Stéphane Gaubert,Ricardo D. Katz,Mateusz Skomra

2024-11-12

Abstract:We develop value iteration-based algorithms to solve in a unified manner different classes of combinatorial zero-sum games with mean-payoff type rewards. These algorithms rely on an oracle, evaluating the dynamic programming operator up to a given precision. We show that the number of calls to the oracle needed to determine exact optimal (positional) strategies is, up to a factor polynomial in the dimension, of order R/sep, where the "separation" sep is defined as the minimal difference between distinct values arising from strategies, and R is a metric estimate, involving the norm of approximate sub and super-eigenvectors of the dynamic programming operator. We illustrate this method by two applications. The first one is a new proof, leading to improved complexity estimates, of a theorem of Boros, Elbassioni, Gurvich and Makino, showing that turn-based mean-payoff games with a fixed number of random positions can be solved in pseudo-polynomial time. The second one concerns entropy games, a model introduced by Asarin, Cervelle, Degorre, Dima, Horn and Kozyakin. The rank of an entropy game is defined as the maximal rank among all the ambiguity matrices determined by strategies of the two players. We show that entropy games with a fixed rank, in their original formulation, can be solved in polynomial time, and that an extension of entropy games incorporating weights can be solved in pseudo-polynomial time under the same fixed rank condition.

Computer Science and Game Theory
Time and the Prisoner's Dilemma

Yishay Mor,Jeffrey S. Rosenschein

DOI: https://doi.org/10.48550/arXiv.cs/0701139

2007-01-22

Abstract:This paper examines the integration of computational complexity into game theoretic models. The example focused on is the Prisoner's Dilemma, repeated for a finite length of time. We show that a minimal bound on the players' computational ability is sufficient to enable cooperative behavior. In addition, a variant of the repeated Prisoner's Dilemma game is suggested, in which players have the choice of opting out. This modification enriches the game and suggests dominance of cooperative strategies. Competitive analysis is suggested as a tool for investigating sub-optimal (but computationally tractable) strategies and game theoretic models in general. Using competitive analysis, it is shown that for bounded players, a sub-optimal strategy might be the optimal choice, given resource limitations.

Computer Science and Game Theory,Artificial Intelligence
Concurrent Stochastic Games with Stateful-discounted and Parity Objectives: Complexity and Algorithms

Ali Asadi,Krishnendu Chatterjee,Raimundo Saona,Jakub Svoboda

2024-10-08

Abstract:We study two-player zero-sum concurrent stochastic games with finite state and action space played for an infinite number of steps. In every step, the two players simultaneously and independently choose an action. Given the current state and the chosen actions, the next state is obtained according to a stochastic transition function. An objective is a measurable function on plays (or infinite trajectories) of the game, and the value for an objective is the maximal expectation that the player can guarantee against the adversarial player. We consider: (a) stateful-discounted objectives, which are similar to the classical discounted-sum objectives, but states are associated with different discount factors rather than a single discount factor; and (b) parity objectives, which are a canonical representation for $\omega$-regular objectives. For stateful-discounted objectives, given an ordering of the discount factors, the limit value is the limit of the value of the stateful-discounted objectives, as the discount factors approach zero according to the given order. The computational problem we consider is the approximation of the value within an arbitrary additive error. The above problem is known to be in EXPSPACE for the limit value of stateful-discounted objectives and in PSPACE for parity objectives. The best-known algorithms for both the above problems are at least exponential time, with an exponential dependence on the number of states and actions. Our main results for the value approximation problem for the limit value of stateful-discounted objectives and parity objectives are as follows: (a) we establish TFNP[NP] complexity; and (b) we present algorithms that improve the dependency on the number of actions in the exponent from linear to logarithmic. In particular, if the number of states is constant, our algorithms run in polynomial time.

Computer Science and Game Theory
A Generic Strategy Improvement Method for Simple Stochastic Games

D. Auger,X. Badin de Montjoye,Y. Strozecki

DOI: https://doi.org/10.48550/arXiv.2102.04922

2021-07-08

Abstract:We present a generic strategy iteration algorithm (GSIA) to find an optimal strategy of a simple stochastic game (SSG). We prove the correctness of GSIA, and derive a general complexity bound, which implies and improves on the results of several articles. First, we remove the assumption that the SSG is stopping, which is usually obtained by a polynomial blowup of the game. Second, we prove a tight bound on the denominator of the values associated to a strategy, and use it to prove that all strategy iteration algorithms are in fact fixed parameter tractable in the number of random vertices. All known strategy iteration algorithms can be seen as instances of GSIA, which allows to analyze the complexity of converge from below by Condon and to propose a class of algorithms generalising Gimbert and Horn's algorithm. These algorithms require less than $r!$ iterations in general and less iterations than the current best deterministic algorithm for binary SSGs given by Ibsen-Jensen and Miltersen.

Computer Science and Game Theory
Strategy Complexity of Concurrent Stochastic Games with Safety and Reachability Objectives

Krishnendu Chatterjee,Kristoffer Arnsfelt Hansen,Rasmus Ibsen-Jensen

DOI: https://doi.org/10.48550/arXiv.1506.02434

2015-06-08

Abstract:We consider finite-state concurrent stochastic games, played by k>=2 players for an infinite number of rounds, where in every round, each player simultaneously and independently of the other players chooses an action, whereafter the successor state is determined by a probability distribution given by the current state and the chosen actions. We consider reachability objectives that given a target set of states require that some state in the target is visited, and the dual safety objectives that given a target set require that only states in the target set are visited. We are interested in the complexity of stationary strategies measured by their patience, which is defined as the inverse of the smallest nonzero probability employed. Our main results are as follows: We show that in two-player zero-sum concurrent stochastic games (with reachability objective for one player and the complementary safety objective for the other player): (i) the optimal bound on the patience of optimal and epsilon-optimal strategies, for both players is doubly exponential; and (ii) even in games with a single nonabsorbing state exponential (in the number of actions) patience is necessary. In general we study the class of non-zero-sum games admitting stationary epsilon-Nash equilibria. We show that if there is at least one player with reachability objective, then doubly-exponential patience may be needed for epsilon-Nash equilibrium strategies, whereas in contrast if all players have safety objectives, the optimal bound on patience for epsilon-Nash equilibrium strategies is only exponential.

Computer Science and Game Theory,Logic in Computer Science
The Complexity of Matching Games: A Survey

Márton Benedek,Péter Biró,Matthew Johnson,Daniël Paulusma,Xin Ye

DOI: https://doi.org/10.1613/jair.1.14281

2023-06-21

Abstract:Matching games naturally generalize assignment games, a well-known class of cooperative games. Interest in matching games has grown recently due to some breakthrough results and new applications. This state-of-the-art survey provides an overview of matching games and extensions, such as $b$-matching games and partitioned matching games; the latter originating from the emerging area of international kidney exchange. In this survey we focus on computational complexity aspects of various game-theoretical solution concepts, such as the core, nucleolus and Shapley value, when the input is restricted to a matching game or one if its variants.

Computer Science and Game Theory,Computational Complexity,Data Structures and Algorithms
Simple Games on Complex Networks

Alexandre Benatti,Luciano da F. Costa

2024-06-22

Abstract:The relationship between topology and dynamics of complex systems has motivated continuing interest from the scientific community. In the present work, we address this interesting topic from the perspective of simple games, involving two teams playing according to a small set of simple rules, taking place on four types of complex networks. Starting from a minimalist game, characterized by full symmetry always leading to ties, four other games are described in progressive order of complexity, taking into account the presence of neighbors as well as strategies. Each of these five games, as well as their specific changes when implemented in four types of networks, are studied in terms of statistics of the total duration of the game as well as the number of victories and ties, with several interesting results that substantiate, in some cases, the importance of the network topology on the respective dynamics. As a subsidiary result, the visualization of relationships between the data elements in terms of coincidence similarity networks allowed a more complete and direct interpretation of the obtained results.

Social and Information Networks,Physics and Society
The Complexities of Random-Turn Hex, Square, and Triangle Games

Yi Yang,Shuigeng Zhou,Jihong Guan,Chuansheng Shen

DOI: https://doi.org/10.1109/tg.2020.3033720

IF: 1.237

2022-01-01

IEEE Transactions on Games

Abstract:In random-turn games, players toss a coin to decide who moves. This article studies the complexities of the algorithms for playing random-turn connection games perfectly on regular tessellations. Our study theoretically shows that there are algorithms playing random-turn hex, square, and triangle perfectly in O(n(9) . 2.618(n)), O(n(9) . 2.746(n)), andO(n(9) . 3.645(n)) time for each move, respectively, where n is the board size. We then implement the perfect-playing algorithm for the random-turn square andmeasure the actual running time it costs for eachmove. We then compute and analyze the game lengths on random-turn square, hex, and triangle and conjecture that the asymptotic complexity of their game lengths are the same. We finally compare the perfect-playing algorithm with the sampling algorithm by competing against each other, and the numbers of their wins and losses are reported.
On the population size in stochastic differential games

Dylan Possamaï,Ludovic Tangpi

2023-10-16

Abstract:Commuters looking for the shortest path to their destinations, the security of networked computers, hedge funds trading on the same stocks, governments and populations acting to mitigate an epidemic, or employers and employees agreeing on a contact, are all examples of (dynamic) stochastic differential games. In essence, game theory deals with the analysis of strategic interactions among multiple decision-makers. The theory has had enormous impact in a wide variety of fields, but its rigorous mathematical analysis is rather recent. It started with the pioneering work of von Neumann and Morgenstern published in 1944. Since then, game theory has taken centre stage in applied mathematics and related areas. Game theory has also played an important role in unsuspected areas: for instance in military applications, when the analysis of guided interceptor missiles in the 1950s motivated the study of games evolving dynamically in time. Such games (when possibly subject to randomness) are called stochastic differential games. Their study started with the work of Issacs, who crucially recognised the importance of (stochastic) control theory in the area. Over the past few decades since Isaacs's work, a rich theory of stochastic differential game has emerged and branched into several directions. This paper will review recent advances in the study of solvability of stochastic differential games, with a focus on a purely probabilistic technique to approach the problem. Unsurprisingly, the number of players involved in the game is a major factor of the analysis. We will explain how the size of the population impacts the analyses and solvability of the problem, and discuss mean field games as well as the convergence of finite player games to mean field games.

Probability,Theoretical Economics,Optimization and Control
Complexity of stochastic dual dynamic programming

Guanghui Lan

DOI: https://doi.org/10.1007/s10107-020-01567-1

IF: 2.7

2020-09-15

Mathematical Programming

Abstract:Stochastic dual dynamic programming is a cutting plane type algorithm for multi-stage stochastic optimization originated about 30 years ago. In spite of its popularity in practice, there does not exist any analysis on the convergence rates of this method. In this paper, we first establish the number of iterations, i.e., iteration complexity, required by a basic dual dynamic programming method for solving single-scenario multi-stage optimization problems, by introducing novel mathematical tools including the saturation of search points. We then refine these basic tools and establish the iteration complexity for an explorative dual dynamic programing method proposed herein and the classic stochastic dual dynamic programming method for solving more general multi-stage stochastic optimization problems under the standard stage-wise independence assumption. Our results indicate that the complexity of these methods mildly increases with the number of stages <i>T</i>, in fact linearly dependent on <i>T</i> for discounted problems. Therefore, they are efficient for strategic decision making which involves a large number of stages, but with a relatively small number of decision variables in each stage. Without explicitly discretizing the state and action spaces, these methods might also be pertinent to the related reinforcement learning and stochastic control areas.

mathematics, applied,computer science, software engineering,operations research & management science
Stochastic Games with Minimally Bounded Action Costs

David Mguni

2024-08-01

Abstract:In many multi-player interactions, players incur strictly positive costs each time they execute actions e.g. 'menu costs' or transaction costs in financial systems. Since acting at each available opportunity would accumulate prohibitively large costs, the resulting decision problem is one in which players must make strategic decisions about when to execute actions in addition to their choice of action. This paper analyses a discrete-time stochastic game (SG) in which players face minimally bounded positive costs for each action and influence the system using impulse controls. We prove SGs of two-sided impulse control have a unique value and characterise the saddle point equilibrium in which the players execute actions at strategically chosen times in accordance with Markovian strategies. We prove the game respects a dynamic programming principle and that the Markov perfect equilibrium can be computed as a limit point of a sequence of Bellman operations. We then introduce a new Q-learning variant which we show converges almost surely to the value of the game enabling solutions to be extracted in unknown settings. Lastly, we extend our results to settings with budgetory constraints.

Multiagent Systems

The Complexity of Simple Stochastic Games

Comparison of Algorithms for Simple Stochastic Games (Full Version)

Comparison of Algorithms for Simple Stochastic Games

On the computational complexity of solving stochastic mean-payoff games

Exact Algorithms for Solving Stochastic Games

Simple Stochastic Stopping Games: A Generator and Benchmark Library

Optimal Time-Abstract Schedulers for CTMDPs and Markov Games

Simplified Game of Life: Algorithms and Complexity

On the complexity of computing Markov perfect equilibrium in general-sum stochastic games

Universal Complexity Bounds Based on Value Iteration for Stochastic Mean Payoff Games and Entropy Games

Time and the Prisoner's Dilemma

Concurrent Stochastic Games with Stateful-discounted and Parity Objectives: Complexity and Algorithms

A Generic Strategy Improvement Method for Simple Stochastic Games

Strategy Complexity of Concurrent Stochastic Games with Safety and Reachability Objectives

The Complexity of Matching Games: A Survey

Simple Games on Complex Networks

The Complexities of Random-Turn Hex, Square, and Triangle Games

On the population size in stochastic differential games

Complexity of stochastic dual dynamic programming

Stochastic Games with Minimally Bounded Action Costs