Abstract:Learning models of the environment from data is often viewed as an essential component to building intelligent reinforcement learning (RL) agents. The common practice is to separate the learning of the model from its use, by constructing a model of the environment's dynamics that correctly predicts the observed state transitions. In this paper we argue that the limited representational resources of model-based RL agents are better used to build models that are directly useful for value-based planning. As our main contribution, we introduce the principle of value equivalence: two models are value equivalent with respect to a set of functions and policies if they yield the same Bellman updates. We propose a formulation of the model learning problem based on the value equivalence principle and analyze how the set of feasible solutions is impacted by the choice of policies and functions. Specifically, we show that, as we augment the set of policies and functions considered, the class of value equivalent models shrinks, until eventually collapsing to a single point corresponding to a model that perfectly describes the environment. In many problems, directly modelling state-to-state transitions may be both difficult and unnecessary. By leveraging the value-equivalence principle one may find simpler models without compromising performance, saving computation and memory. We illustrate the benefits of value-equivalent model learning with experiments comparing it against more traditional counterparts like maximum likelihood estimation. More generally, we argue that the principle of value equivalence underlies a number of recent empirical successes in RL, such as Value Iteration Networks, the Predictron, Value Prediction Networks, TreeQN, and MuZero, and provides a first theoretical underpinning of those results.

Model-based value estimation for efficient model-free reinforcement learning

Model-Assisted Reinforcement Learning with Adaptive Ensemble Value Expansion

Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learning

Improving Probability Estimation Through Active Probabilistic Model Learning.

Model predictive control-based value estimation for efficient reinforcement learning

Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion

Dynamic Horizon Value Estimation for Model-based Reinforcement Learning

Model-Based Robot Learning Control with Uncertainty Directed Exploration

Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Function

Model-advantage and value-aware models for model-based reinforcement learning: bridging the gap in theory and practice

Dynamic-Horizon Model-Based Value Estimation With Latent Imagination

Model-based Reinforcement Learning with Multi-step Plan Value Estimation

Efficient Exploration in Continuous-time Model-based Reinforcement Learning

Between Rate-Distortion Theory & Value Equivalence in Model-Based Reinforcement Learning

The Value Equivalence Principle for Model-Based Reinforcement Learning

Blending MPC & Value Function Approximation for Efficient Reinforcement Learning

Foresee then Evaluate: Decomposing Value Estimation with Latent Future Prediction.

Optimal State Estimation Using Model-Free Reinforcement Learning

Model-Based Reinforcement Learning via Meta-Policy Optimization

Value Gradient weighted Model-Based Reinforcement Learning

Imagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models