Abstract:Offline reinforcement learning (RL) refers to the problem of learning policies entirely from a large batch of previously collected data. This problem setting offers the promise of utilizing such datasets to acquire policies without any costly or dangerous active exploration. However, it is also challenging, due to the distributional shift between the offline training data and those states visited by the learned policy. Despite significant recent progress, the most successful prior methods are model-free and constrain the policy to the support of data, precluding generalization to unseen states. In this paper, we first observe that an existing model-based RL algorithm already produces significant gains in the offline setting compared to model-free approaches. However, standard model-based RL methods, designed for the online setting, do not provide an explicit mechanism to avoid the offline setting's distributional shift issue. Instead, we propose to modify the existing model-based RL methods by applying them with rewards artificially penalized by the uncertainty of the dynamics. We theoretically show that the algorithm maximizes a lower bound of the policy's return under the true MDP. We also characterize the trade-off between the gain and risk of leaving the support of the batch data. Our algorithm, Model-based Offline Policy Optimization (MOPO), outperforms standard model-based RL algorithms and prior state-of-the-art model-free offline RL algorithms on existing offline RL benchmarks and two challenging continuous control tasks that require generalizing from data collected for a different task. The code is available at <a class="link-external link-https" href="https://github.com/tianheyu927/mopo" rel="external noopener nofollow">this https URL</a>.

Model-based Offline Reinforcement Learning with Count-based Conservatism

DROP: Conservative Model-based Optimization for Offline Reinforcement Learning

Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization

CROP: Conservative Reward for Model-based Offline Policy Optimization

Efficient Offline Reinforcement Learning With Relaxed Conservatism

Offline Reinforcement Learning with Reverse Model-based Imagination

DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learning

MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator

OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning

Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learning

Mildly Conservative Q-Learning for Offline Reinforcement Learning

MOPO: Model-based Offline Policy Optimization

Strategically Conservative Q-Learning

Model-Based Offline Policy Optimization with Distribution Correcting Regularization.

MOReL : Model-Based Offline Reinforcement Learning

Settling the Sample Complexity of Model-Based Offline Reinforcement Learning

DCE: Offline Reinforcement Learning with Double Conservative Estimates

Model-Bellman Inconsistency for Model-based Offline Reinforcement Learning.

Offline Model-Based Reinforcement Learning with Anti-Exploration

CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning

COMBO: Conservative Offline Model-Based Policy Optimization