Abstract:Models of many real-life applications, such as queuing models of communication networks or computing systems, have a countably infinite state-space. Algorithmic and learning procedures that have been developed to produce optimal policies mainly focus on finite state settings, and do not directly apply to these models. To overcome this lacuna, in this work we study the problem of optimal control of a family of discrete-time countable state-space Markov Decision Processes (MDPs) governed by an unknown parameter $\theta\in\Theta$, and defined on a countably-infinite state space $\mathcal X=\mathbb{Z}_+^d$, with finite action space $\mathcal A$, and an unbounded cost function. We take a Bayesian perspective with the random unknown parameter $\boldsymbol{\theta}^*$ generated via a given fixed prior distribution on $\Theta$. To optimally control the unknown MDP, we propose an algorithm based on Thompson sampling with dynamically-sized episodes: at the beginning of each episode, the posterior distribution formed via Bayes' rule is used to produce a parameter estimate, which then decides the policy applied during the episode. To ensure the stability of the Markov chain obtained by following the policy chosen for each parameter, we impose ergodicity assumptions. From this condition and using the solution of the average cost Bellman equation, we establish an $\tilde O(dh^d\sqrt{|\mathcal A|T})$ upper bound on the Bayesian regret of our algorithm, where $T$ is the time-horizon. Finally, to elucidate the applicability of our algorithm, we consider two different queuing models with unknown dynamics, and show that our algorithm can be applied to develop approximately optimal control algorithms.

Absorbing Markov Decision Processes

A note on weak compactness of occupation measures for an absorbing Markov decision process

Extreme Occupation Measures in Markov Decision Processes with an Absorbing State

Further remarks on absorbing Markov decision processes

On the Continuity of the Projection Mapping from Strategic Measures to Occupation Measures in Absorbing Markov Decision Processes

Extreme occupation measures in Markov decision processes with a cemetery

Maximizing the probability of visiting a set infinitely often for a Markov decision process with Borel state and action spaces

Measurized Markov Decision Processes Part I: The Discounted Infinite Horizon Criterion

Bayesian Learning of Optimal Policies in Markov Decision Processes with Countably Infinite State-Space

Intermittently Observable Markov Decision Processes

Discrete Type Shock Semi-Markov Decision Processes with Borel State Space

A Lazy Abstraction Algorithm for Markov Decision Processes: Theory and Initial Evaluation

Markov Decision Processes with Incomplete Information and Semi-Uniform Feller Transition Probabilities

Primal-Dual Regression Approach for Markov Decision Processes with General State and Action Spaces

Continuous-Time Mean Field Markov Decision Models

Relaxed Equilibria for Time-Inconsistent Markov Decision Processes

POMDPs in Continuous Time and Discrete Spaces

Numerical method to solve impulse control problems for partially observed piecewise deterministic Markov processes

Robust Markov Decision Processes: A Place Where AI and Formal Methods Meet

The Finiteness of the Reward Function and the Optimal Value Function in Markov Decision Processes