Abstract:Models of many real-life applications, such as queuing models of communication networks or computing systems, have a countably infinite state-space. Algorithmic and learning procedures that have been developed to produce optimal policies mainly focus on finite state settings, and do not directly apply to these models. To overcome this lacuna, in this work we study the problem of optimal control of a family of discrete-time countable state-space Markov Decision Processes (MDPs) governed by an unknown parameter $\theta\in\Theta$, and defined on a countably-infinite state space $\mathcal X=\mathbb{Z}_+^d$, with finite action space $\mathcal A$, and an unbounded cost function. We take a Bayesian perspective with the random unknown parameter $\boldsymbol{\theta}^*$ generated via a given fixed prior distribution on $\Theta$. To optimally control the unknown MDP, we propose an algorithm based on Thompson sampling with dynamically-sized episodes: at the beginning of each episode, the posterior distribution formed via Bayes' rule is used to produce a parameter estimate, which then decides the policy applied during the episode. To ensure the stability of the Markov chain obtained by following the policy chosen for each parameter, we impose ergodicity assumptions. From this condition and using the solution of the average cost Bellman equation, we establish an $\tilde O(dh^d\sqrt{|\mathcal A|T})$ upper bound on the Bayesian regret of our algorithm, where $T$ is the time-horizon. Finally, to elucidate the applicability of our algorithm, we consider two different queuing models with unknown dynamics, and show that our algorithm can be applied to develop approximately optimal control algorithms.

Maximizing the probability of visiting a set infinitely often for a Markov decision process with Borel state and action spaces

Risk-sensitive Continuous-Time Markov Decision Processes with Unbounded Rates and Borel Spaces

Average Optimality for Markov Decision Processes in Borel Spaces: a New Condition and Approach

Bayesian Learning of Optimal Policies in Markov Decision Processes with Countably Infinite State-Space

Extreme Occupation Measures in Markov Decision Processes with an Absorbing State

Continuous-Time Markov Decision Processes with Unbounded Transition and Discounted-Reward Rates

On Strategic Measures and Optimality Properties in Discrete-Time Stochastic Control with Universally Measurable Policies

Risk-sensitive discounted Markov decision processes with unbounded reward functions and Borel spaces

Mean Field Markov Decision Processes

Average Optimality For Continuous-Time Markov Decision Processes In Polish Spaces

Absorbing Markov Decision Processes

Asymptotically Optimal Policies for Weakly Coupled Markov Decision Processes

A note on weak compactness of occupation measures for an absorbing Markov decision process

New Average Optimality Conditions for Semi-Markov Decision Processes in Borel Spaces.

Continuous-Time Controlled Markov Chains with Discounted Rewards

Another Set of Verifiable Conditions for Average Markov Decision Processes with Borel Spaces

The minimal hitting probability of continuous-time controlled Markov systems with countable states

On Borkar and Young Relaxed Control Topologies and Continuous Dependence of Invariant Measures on Control Policy

Constrained Markov Decision Processes with Non-constant Discount Factor

Risk Probability Minimization Problems for Continuous-Time Markov Decision Processes on Finite Horizon

Optimal models with maximizing probability of first achieving target value in the preceding stages