Abstract:Learning models of the environment from data is often viewed as an essential component to building intelligent reinforcement learning (RL) agents. The common practice is to separate the learning of the model from its use, by constructing a model of the environment's dynamics that correctly predicts the observed state transitions. In this paper we argue that the limited representational resources of model-based RL agents are better used to build models that are directly useful for value-based planning. As our main contribution, we introduce the principle of value equivalence: two models are value equivalent with respect to a set of functions and policies if they yield the same Bellman updates. We propose a formulation of the model learning problem based on the value equivalence principle and analyze how the set of feasible solutions is impacted by the choice of policies and functions. Specifically, we show that, as we augment the set of policies and functions considered, the class of value equivalent models shrinks, until eventually collapsing to a single point corresponding to a model that perfectly describes the environment. In many problems, directly modelling state-to-state transitions may be both difficult and unnecessary. By leveraging the value-equivalence principle one may find simpler models without compromising performance, saving computation and memory. We illustrate the benefits of value-equivalent model learning with experiments comparing it against more traditional counterparts like maximum likelihood estimation. More generally, we argue that the principle of value equivalence underlies a number of recent empirical successes in RL, such as Value Iteration Networks, the Predictron, Value Prediction Networks, TreeQN, and MuZero, and provides a first theoretical underpinning of those results.

Control-Oriented Model-Based Reinforcement Learning with Implicit Differentiation

Actively Learning Reinforcement Learning: A Stochastic Optimal Control Approach

Model-Based Reinforcement Learning via Stochastic Hybrid Models

The Value Equivalence Principle for Model-Based Reinforcement Learning

Model-Augmented Actor-Critic: Backpropagating through Paths

Implicit Posterior Sampling Reinforcement Learning for Continuous Control

Value-Biased Maximum Likelihood Estimation for Model-based Reinforcement Learning in Discounted Linear MDPs

Blending MPC & Value Function Approximation for Efficient Reinforcement Learning

Policy-Based Bayesian Experimental Design for Non-Differentiable Implicit Models

Between Rate-Distortion Theory & Value Equivalence in Model-Based Reinforcement Learning

Model-Based Reinforcement Learning via Meta-Policy Optimization

Model-based reinforcement learning with dimension reduction

Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees

Efficient Exploration in Continuous-time Model-based Reinforcement Learning

Bidirectional Model-based Policy Optimization

Model-Based Reinforcement Learning Via Imagination with Derived Memory.

A Model-Based Approach for Improving Reinforcement Learning Efficiency Leveraging Expert Observations

Maximum Entropy Model-based Reinforcement Learning

Value Gradient weighted Model-Based Reinforcement Learning

Deep Incremental Model Based Reinforcement Learning: A One-Step Lookback Approach for Continuous Robotics Control

Differentiable MPC for End-to-end Planning and Control