Abstract:We consider the problem of learning in a non-stationary reinforcement learning (RL) environment, where the setting can be fully described by a piecewise stationary discrete-time Markov decision process (MDP). We introduce a variant of the Restarted Bayesian Online Change-Point Detection algorithm (R-BOCPD) that operates on input streams originating from the more general multinomial distribution and provides near-optimal theoretical guarantees in terms of false-alarm rate and detection delay. Based on this, we propose an improved version of the UCRL2 algorithm for MDPs with state transition kernel sampled from a multinomial distribution, which we call R-BOCPD-UCRL2. We perform a finite-time performance analysis and show that R-BOCPD-UCRL2 enjoys a favorable regret bound of $O\left(D O \sqrt{A T K_T \log\left (\frac{T}{\delta} \right) + \frac{K_T \log \frac{K_T}{\delta}}{\min\limits_\ell \: \mathbf{KL}\left( {\mathbf{\theta}^{(\ell+1)}}\mid\mid{\mathbf{\theta}^{(\ell)}}\right)}}\right)$, where $D$ is the largest MDP diameter from the set of MDPs defining the piecewise stationary MDP setting, $O$ is the finite number of states (constant over all changes), $A$ is the finite number of actions (constant over all changes), $K_T$ is the number of change points up to horizon $T$, and $\mathbf{\theta}^{(\ell)}$ is the transition kernel during the interval $[c_\ell, c_{\ell+1})$, which we assume to be multinomially distributed over the set of states $\mathbb{O}$. Interestingly, the performance bound does not directly scale with the variation in MDP state transition distributions and rewards, ie. can also model abrupt changes. In practice, R-BOCPD-UCRL2 outperforms the state-of-the-art in a variety of scenarios in synthetic environments. We provide a detailed experimental setup along with a code repository (upon publication) that can be used to easily reproduce our experiments.

BLPA: Bayesian Learn-Predict-adjust Method for Online Detection of Recurrent Changepoints.

Change Point Detection with Neural Online Density-Ratio Estimator

Online Change-Point Detection of Linear Regression Models

Bayesian Autoregressive Online Change-Point Detection with Time-Varying Parameters

Bayesian changepoint detection via logistic regression and the topological analysis of image series

Restarted Bayesian Online Change-point Detection for Non-Stationary Markov Decision Processes

Automatic Change-Point Detection in Time Series via Deep Learning

A Bayesian Partially Observable Online Change Detection Approach with Thompson Sampling

Online Bayesian changepoint detection for network Poisson processes with community structure

Online Change Points Detection for Linear Dynamical Systems with Finite Sample Guarantees

Detecting weak changes in dynamic events over networks

Sequential Changepoint Detection in Neural Networks with Checkpoints

Online Structural Change-Point Detection of High-dimensional Streaming Data via Dynamic Sparse Subspace Learning

An Online Bayesian Approach to Change-Point Detection for Categorical Data

Memory-free Online Change-point Detection: A Novel Neural Network Approach

Subspace Change-Point Detection: A New Model and Solution.

Leveraging change point detection to discover natural experiments in data

Latent Neural Stochastic Differential Equations for Change Point Detection

Sequential Kalman filter for fast online changepoint detection in longitudinal health records

Doubly-online changepoint detection for monitoring health status during sports activities

Non-segmental Bayesian Detection of Multiple Change-points