Abstract:In this paper, we consider a large class of constrained non-cooperative stochastic Markov games with countable state spaces and discounted cost criteria. In one-player case, i.e., constrained discounted Markov decision models, it is possible to formulate a static optimisation problem whose solution determines a stationary optimal strategy (alias control or policy) in the dynamical infinite horizon model. This solution lies in the compact convex set of all occupation measures induced by strategies, defined on the set of state-action pairs. In case of n-person discounted games the occupation measures are induced by strategies of all players. Therefore, it is difficult to generalise the approach for constrained discounted Markov decision processes directly. It is not clear how to define the domain for the best-response correspondence whose fixed point induces a stationary equilibrium in the Markov game. This domain should be the Cartesian product of compact convex sets in locally convex topological vector spaces. One of our main results shows how to overcome this difficulty and define a constrained non-cooperative static game whose Nash equilibrium induces by a stationary Nash equilibrium in the Markov game. This is done for games with bounded cost functions and positive initial state distribution. An extension to a class of Markov games with unbounded costs and arbitrary initial state distribution relies on approximation of the unbounded game by bounded ones with positive initial state distributions. In the unbounded case, we assume the uniform integrability of the discounted costs with respect to all probability measures induced by strategies of the players, defined on the space of plays (histories) of the game. Our assumptions are weaker than those applied in earlier works on discounted dynamic programming or stochastic games using so-called weighted norm approaches.

Long Run Stochastic Control Problems with General Discounting

Long-Run Impulse Control with Generalized Discounting

On Average Risk-sensitive Markov Control Processes

Smooth solutions to discounted reward control problems with unbounded discount rate and financial applications

Risk-sensitive Markov control processes

Stability of long run functionals with respect to stationary Markov controls

Stochastic control up to a hitting time: optimality and rolling-horizon implementation

Blackwell optimality and policy stability for long-run risk sensitive stochastic control

Discrete-Time Approximation of Risk-Averse Control Problems for Diffusion Processes

Discrete time risk sensitive control problem

Risk Aware Minimum Principle for Optimal Control of Stochastic Differential Equations

Continuous Time Markov Decision Processes with Expected Discounted Total Rewards

A Discount Vanishing Approximation for Markov Decision Processes with Risk Sensitivity

Stochastic Near-Optimal Controls for Path-Dependent Systems

Refraction strategies in stochastic control: optimality for a general Lévy process model

Near-Optimal Control for Time-Varying Linear Discrete Systems with Additive Nonlinearities and Random Gains

Constrained discounted stochastic games

On the hierarchical risk-averse control problems for diffusion processes

De Finetti's control problem with a concave bound on the control rate

An approach for regime-switching stochastic control problems with memory and terminal conditions

Optimal Stopping for Strong Markov Processes: Explicit solutions and verification theorems for diffusions, multidimensional diffusions, and jump-processes