Abstract:In this paper, we consider a large class of constrained non-cooperative stochastic Markov games with countable state spaces and discounted cost criteria. In one-player case, i.e., constrained discounted Markov decision models, it is possible to formulate a static optimisation problem whose solution determines a stationary optimal strategy (alias control or policy) in the dynamical infinite horizon model. This solution lies in the compact convex set of all occupation measures induced by strategies, defined on the set of state-action pairs. In case of n-person discounted games the occupation measures are induced by strategies of all players. Therefore, it is difficult to generalise the approach for constrained discounted Markov decision processes directly. It is not clear how to define the domain for the best-response correspondence whose fixed point induces a stationary equilibrium in the Markov game. This domain should be the Cartesian product of compact convex sets in locally convex topological vector spaces. One of our main results shows how to overcome this difficulty and define a constrained non-cooperative static game whose Nash equilibrium induces by a stationary Nash equilibrium in the Markov game. This is done for games with bounded cost functions and positive initial state distribution. An extension to a class of Markov games with unbounded costs and arbitrary initial state distribution relies on approximation of the unbounded game by bounded ones with positive initial state distributions. In the unbounded case, we assume the uniform integrability of the discounted costs with respect to all probability measures induced by strategies of the players, defined on the space of plays (histories) of the game. Our assumptions are weaker than those applied in earlier works on discounted dynamic programming or stochastic games using so-called weighted norm approaches.

Smooth solutions to discounted reward control problems with unbounded discount rate and financial applications

Long Run Stochastic Control Problems with General Discounting

Long-Run Impulse Control with Generalized Discounting

Weak equilibria for time-inconsistent control: with applications to investment-withdrawal decisions

The Maximum Principle for Discounted Optimal Control of Partially Observed Forward-Backward Stochastic Systems with Jumps on Infinite Horizon

Stochastic control problems with state-reflections arising from relaxed benchmark tracking

Solutions to Equilibrium HJB Equations for Time-Inconsistent Deterministic Linear Quadratic Control: Characterization and Uniqueness

The Solution to an Impulse Control Problem Motivated by Optimal Harvesting

Equilibria for Time-inconsistent Singular Control Problems

Stochastic Control Problems with State Reflections Arising from Relaxed Benchmark Tracking

Discrete-Time Hybrid Control Processes with Unbounded Costs

Impulse Control of Multi-dimensional Jump Diffusions in Finite Time Horizon

Continuous Time Markov Decision Processes with Expected Discounted Total Rewards

A zero-sum hybrid stochastic differential game with impulse controls

Stochastic control up to a hitting time: optimality and rolling-horizon implementation

Weak Equilibriums for Time-Inconsistent Stopping Control Problems, with Applications to Investment-Withdrawal Decision Model

A zero-sum hybrid stochastic differential game with switching controls

Discrete-Time Approximation of Risk-Averse Control Problems for Diffusion Processes

Constrained discounted stochastic games

Equilibrium control theory for Kihlstrom-Mirman preferences in continuous time

Equilibrium master equations for time-inconsistent problems with distribution dependent rewards