Abstract:Learning a transition model via Maximum Likelihood Estimation (MLE) followed by planning inside the learned model is perhaps the most standard and simplest Model-based Reinforcement Learning (RL) framework. In this work, we show that such a simple Model-based RL scheme, when equipped with optimistic and pessimistic planning procedures, achieves strong regret and sample complexity bounds in online and offline RL settings. Particularly, we demonstrate that under the conditions where the trajectory-wise reward is normalized between zero and one and the transition is time-homogenous, it achieves nearly horizon-free and second-order bounds. Nearly horizon-free means that our bounds have no polynomial dependence on the horizon of the Markov Decision Process. A second-order bound is a type of instance-dependent bound that scales with respect to the variances of the returns of the policies which can be small when the system is nearly deterministic and (or) the optimal policy has small values. We highlight that our algorithms are simple, fairly standard, and indeed have been extensively studied in the RL literature: they learn a model via MLE, build a version space around the MLE solution, and perform optimistic or pessimistic planning depending on whether operating in the online or offline mode. These algorithms do not rely on additional specialized algorithmic designs such as learning variances and performing variance-weighted learning and thus can easily leverage non-linear function approximations. The simplicity of the algorithms also implies that our horizon-free and second-order regret analysis is actually standard and mainly follows the general framework of optimism/pessimism in the face of uncertainty.

Nearly Minimax Optimal Reward-free Reinforcement Learning.

Reward-Free Exploration for Reinforcement Learning

Successive Convex Approximation Based Off-Policy Optimization for Constrained Reinforcement Learning

Towards Minimax Optimal Reward-free Reinforcement Learning in Linear MDPs

Reward-Free Model-Based Reinforcement Learning with Linear Function Approximation

Optimal Horizon-Free Reward-Free Exploration for Linear Mixture MDPs

Nearly Horizon-Free Offline Reinforcement Learning

Breaking the sample complexity barrier to regret-optimal model-free reinforcement learning

Multiple Suboptimal Policies Integrated Reinforcement Learning Algorithm for Path Planning

Near-Optimal Regret Bounds for Multi-batch Reinforcement Learning

Nearly Minimax Optimal Reinforcement Learning with Linear Function Approximation.

Improved Sample Complexity for Reward-free Reinforcement Learning under Low-rank MDPs

Overcoming the Long Horizon Barrier for Sample-Efficient Reinforcement Learning with Latent Low-Rank Structure

Near-Optimal Reward-Free Exploration for Linear Mixture MDPs with Plug-in Solver.

Minimax-Optimal Reward-Agnostic Exploration in Reinforcement Learning

Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Game

Exploration by Maximizing Renyi Entropy for Reward-Free RL Framework.

Model-Free Reinforcement Learning: from Clipped Pseudo-Regret to Sample Complexity

A Stochastic Trust-Region Framework for Policy Optimization

Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processes

Model-based RL as a Minimalist Approach to Horizon-Free and Second-Order Bounds