Abstract:Offline reinforcement learning (RL) refers to the problem of learning policies entirely from a large batch of previously collected data. This problem setting offers the promise of utilizing such datasets to acquire policies without any costly or dangerous active exploration. However, it is also challenging, due to the distributional shift between the offline training data and those states visited by the learned policy. Despite significant recent progress, the most successful prior methods are model-free and constrain the policy to the support of data, precluding generalization to unseen states. In this paper, we first observe that an existing model-based RL algorithm already produces significant gains in the offline setting compared to model-free approaches. However, standard model-based RL methods, designed for the online setting, do not provide an explicit mechanism to avoid the offline setting's distributional shift issue. Instead, we propose to modify the existing model-based RL methods by applying them with rewards artificially penalized by the uncertainty of the dynamics. We theoretically show that the algorithm maximizes a lower bound of the policy's return under the true MDP. We also characterize the trade-off between the gain and risk of leaving the support of the batch data. Our algorithm, Model-based Offline Policy Optimization (MOPO), outperforms standard model-based RL algorithms and prior state-of-the-art model-free offline RL algorithms on existing offline RL benchmarks and two challenging continuous control tasks that require generalizing from data collected for a different task. The code is available at <a class="link-external link-https" href="https://github.com/tianheyu927/mopo" rel="external noopener nofollow">this https URL</a>.

Model-Based Offline Policy Optimization with Distribution Correcting Regularization.

DROP: Conservative Model-based Optimization for Offline Reinforcement Learning

Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimization

Behavior Proximal Policy Optimization

Beyond Reward: Offline Preference-guided Policy Optimization

Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning

CROP: Conservative Reward for Model-based Offline Policy Optimization

MOPO: Model-based Offline Policy Optimization

Model-Based Offline Adaptive Policy Optimization with Episodic Memory

Model-Based Offline Meta-Reinforcement Learning with Regularization

Optimistic Model Rollouts for Pessimistic Offline Policy Optimization

Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage

Boosting Offline Reinforcement Learning via Data Rebalancing

Policy Regularization with Dataset Constraint for Offline Reinforcement Learning

Model-Based Offline Weighted Policy Optimization (Student Abstract)

Federated Offline Policy Optimization with Dual Regularization

Diversification of Adaptive Policy for Effective Offline Reinforcement Learning

Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage

Enhancing OOD Generalization in Offline Reinforcement Learning with Energy-Based Policy Optimization

Iteratively Refined Behavior Regularization for Offline Reinforcement Learning