Abstract:We consider the problem of learning in a non-stationary reinforcement learning (RL) environment, where the setting can be fully described by a piecewise stationary discrete-time Markov decision process (MDP). We introduce a variant of the Restarted Bayesian Online Change-Point Detection algorithm (R-BOCPD) that operates on input streams originating from the more general multinomial distribution and provides near-optimal theoretical guarantees in terms of false-alarm rate and detection delay. Based on this, we propose an improved version of the UCRL2 algorithm for MDPs with state transition kernel sampled from a multinomial distribution, which we call R-BOCPD-UCRL2. We perform a finite-time performance analysis and show that R-BOCPD-UCRL2 enjoys a favorable regret bound of $O\left(D O \sqrt{A T K_T \log\left (\frac{T}{\delta} \right) + \frac{K_T \log \frac{K_T}{\delta}}{\min\limits_\ell \: \mathbf{KL}\left( {\mathbf{\theta}^{(\ell+1)}}\mid\mid{\mathbf{\theta}^{(\ell)}}\right)}}\right)$, where $D$ is the largest MDP diameter from the set of MDPs defining the piecewise stationary MDP setting, $O$ is the finite number of states (constant over all changes), $A$ is the finite number of actions (constant over all changes), $K_T$ is the number of change points up to horizon $T$, and $\mathbf{\theta}^{(\ell)}$ is the transition kernel during the interval $[c_\ell, c_{\ell+1})$, which we assume to be multinomially distributed over the set of states $\mathbb{O}$. Interestingly, the performance bound does not directly scale with the variation in MDP state transition distributions and rewards, ie. can also model abrupt changes. In practice, R-BOCPD-UCRL2 outperforms the state-of-the-art in a variety of scenarios in synthetic environments. We provide a detailed experimental setup along with a code repository (upon publication) that can be used to easily reproduce our experiments.

Nonparametric Bayesian online change point detection using kernel density estimation with nonparametric hazard function

Change Point Detection for Nonparametric Regression under Strongly Mixing Process

Change Point Detection with Neural Online Density-Ratio Estimator

Nonparametric sequential change-point detection for multivariate time series based on empirical distribution functions

Bayesian nonparametric change point detection for multivariate time series with missing observations

Bayesian Autoregressive Online Change-Point Detection with Time-Varying Parameters

An Online Bayesian Approach to Change-Point Detection for Categorical Data

Confirmatory Bayesian Online Change Point Detection in the Covariance Structure of Gaussian Processes

Online Change-Point Detection of Linear Regression Models

Online Kernel CUSUM for Change-Point Detection

Nonparametric Change-Point Detection for Profiles with Binary Data

Reproduction of scan B-statistic for kernel change-point detection algorithm

Hierarchical Change-Point Detection for Multivariate Time Series via a Ball Detection Function

Robust multiple change-point detection for multivariate variability using data depth

Greedy online change point detection

Non-asymptotic inference for multivariate change point detection

Online Graph-Based Change-Point Detection for High Dimensional Data

Wavelet Analysis of Change Points in Nonparametric Hazard Rate Models under Random Censorship

An encoding approach for stable change point detection

Kernel Change-point Detection with Auxiliary Deep Generative Models

Restarted Bayesian Online Change-point Detection for Non-Stationary Markov Decision Processes