Abstract:Models of many real-life applications, such as queuing models of communication networks or computing systems, have a countably infinite state-space. Algorithmic and learning procedures that have been developed to produce optimal policies mainly focus on finite state settings, and do not directly apply to these models. To overcome this lacuna, in this work we study the problem of optimal control of a family of discrete-time countable state-space Markov Decision Processes (MDPs) governed by an unknown parameter $\theta\in\Theta$, and defined on a countably-infinite state space $\mathcal X=\mathbb{Z}_+^d$, with finite action space $\mathcal A$, and an unbounded cost function. We take a Bayesian perspective with the random unknown parameter $\boldsymbol{\theta}^*$ generated via a given fixed prior distribution on $\Theta$. To optimally control the unknown MDP, we propose an algorithm based on Thompson sampling with dynamically-sized episodes: at the beginning of each episode, the posterior distribution formed via Bayes' rule is used to produce a parameter estimate, which then decides the policy applied during the episode. To ensure the stability of the Markov chain obtained by following the policy chosen for each parameter, we impose ergodicity assumptions. From this condition and using the solution of the average cost Bellman equation, we establish an $\tilde O(dh^d\sqrt{|\mathcal A|T})$ upper bound on the Bayesian regret of our algorithm, where $T$ is the time-horizon. Finally, to elucidate the applicability of our algorithm, we consider two different queuing models with unknown dynamics, and show that our algorithm can be applied to develop approximately optimal control algorithms.

Palm problems arising in BAR approach and its applications

The BAR approach for multiclass queueing networks with SBP service policies

On the Palm distribution of superposition of point processes

Non-parametric asymptotic statistics for the Palm mark distribution of β-mixing marked point processes

Bayesian inference for double Pareto lognormal queues

Diffusion Approximations for Self-excited Systems with Applications to General Branching Processes

Self-similar co-ascent processes and Palm calculus

Long time behavior of semi-Markov modulated perpetuity and some related processes

A Palm Space Approach to Non-Linear Hawkes Processes

Bayesian inference for Matérn repulsive processes

Multi-level reflecting Brownian motion on the half line and its stationary distribution

Asymptotic goodness-of-fit tests for the Palm mark distribution of stationary point processes with correlated marks

Continuous-state branching processes with collisions: First passage times and duality

Poisson process approximation: From Palm theory to Stein's method

Mixing Times and Structural Inference for Bernoulli Autoregressive Processes

Bayesian Learning of Optimal Policies in Markov Decision Processes with Countably Infinite State-Space

Fitting procedure for the two-state Batch Markov modulated Poisson process

On Brownian motion approximation of compound Poisson processes with applications to threshold models

Stick-Breaking processes, Clumping, and Markov Chain Occupation Laws

A modified Conway–Maxwell–Poisson type binomial distribution and its applications