Abstract:Models of many real-life applications, such as queuing models of communication networks or computing systems, have a countably infinite state-space. Algorithmic and learning procedures that have been developed to produce optimal policies mainly focus on finite state settings, and do not directly apply to these models. To overcome this lacuna, in this work we study the problem of optimal control of a family of discrete-time countable state-space Markov Decision Processes (MDPs) governed by an unknown parameter $\theta\in\Theta$, and defined on a countably-infinite state space $\mathcal X=\mathbb{Z}_+^d$, with finite action space $\mathcal A$, and an unbounded cost function. We take a Bayesian perspective with the random unknown parameter $\boldsymbol{\theta}^*$ generated via a given fixed prior distribution on $\Theta$. To optimally control the unknown MDP, we propose an algorithm based on Thompson sampling with dynamically-sized episodes: at the beginning of each episode, the posterior distribution formed via Bayes' rule is used to produce a parameter estimate, which then decides the policy applied during the episode. To ensure the stability of the Markov chain obtained by following the policy chosen for each parameter, we impose ergodicity assumptions. From this condition and using the solution of the average cost Bellman equation, we establish an $\tilde O(dh^d\sqrt{|\mathcal A|T})$ upper bound on the Bayesian regret of our algorithm, where $T$ is the time-horizon. Finally, to elucidate the applicability of our algorithm, we consider two different queuing models with unknown dynamics, and show that our algorithm can be applied to develop approximately optimal control algorithms.

Exact finite approximations of average-cost countable Markov Decision Processes

Approximating Euclidean by Imprecise Markov Decision Processes

Strong Approximation for A Class of Stationary Processes

Discretized Approximations for POMDP with Average Cost

Strong Approximation for Moving Average Processes under Dependence Assumptions

Approximation methods for piecewise deterministic Markov processes and their costs

Risk-Sensitive Average Markov Decision Processes in General Spaces

On an Approximation of Average Cost per Unit Time Impulse Control of Markov Processes

Model approximation in MDPs with unbounded per-step cost

Approximating Markov Processes Through Filtration

On Linear Programming for Constrained and Unconstrained Average-Cost Markov Decision Processes with Countable Action Spaces and Strictly Unbounded Costs

Solution to the risk-sensitive average cost optimality equation in a class of Markov decision processes with finite state space

Average-Cost MDPs with Infinite State and Action Sets: New Sufficient Conditions for Optimality Inequalities and Equations

Characterization of the optimal average cost in Markov decision chains driven by a risk-seeking controller

Average cost optimal control under weak ergodicity hypotheses: Relative value iterations

A Discount Vanishing Approximation for Markov Decision Processes with Risk Sensitivity

Bayesian Learning of Optimal Policies in Markov Decision Processes with Countably Infinite State-Space

Optimal approximating Markov chains for Bayesian inference

Optimal Stationary Policies for a Class of Countable Markov Control Processes

A Unified Approach to Time-Aggregated Markov Decision Processes.

Local Poisson Equations Associated with Discrete-Time Markov Control Processes