Abstract:Offline reinforcement learning (RL) refers to the problem of learning policies entirely from a large batch of previously collected data. This problem setting offers the promise of utilizing such datasets to acquire policies without any costly or dangerous active exploration. However, it is also challenging, due to the distributional shift between the offline training data and those states visited by the learned policy. Despite significant recent progress, the most successful prior methods are model-free and constrain the policy to the support of data, precluding generalization to unseen states. In this paper, we first observe that an existing model-based RL algorithm already produces significant gains in the offline setting compared to model-free approaches. However, standard model-based RL methods, designed for the online setting, do not provide an explicit mechanism to avoid the offline setting's distributional shift issue. Instead, we propose to modify the existing model-based RL methods by applying them with rewards artificially penalized by the uncertainty of the dynamics. We theoretically show that the algorithm maximizes a lower bound of the policy's return under the true MDP. We also characterize the trade-off between the gain and risk of leaving the support of the batch data. Our algorithm, Model-based Offline Policy Optimization (MOPO), outperforms standard model-based RL algorithms and prior state-of-the-art model-free offline RL algorithms on existing offline RL benchmarks and two challenging continuous control tasks that require generalizing from data collected for a different task. The code is available at <a class="link-external link-https" href="https://github.com/tianheyu927/mopo" rel="external noopener nofollow">this https URL</a>.

Behavior Proximal Policy Optimization

Beyond Reward: Offline Preference-guided Policy Optimization

Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimization

DROP: Conservative Model-based Optimization for Offline Reinforcement Learning

Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL

MOPO: Model-based Offline Policy Optimization

PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning

Supported Policy Optimization for Offline Reinforcement Learning

A Non-Monolithic Policy Approach of Offline-to-Online Reinforcement Learning

Offline Reinforcement Learning with Closed-Form Policy Improvement Operators

A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective

Reward-free Offline Reinforcement Learning: Optimizing Behavior Policy Via Action Exploration

Adaptive Policy Learning for Offline-to-Online Reinforcement Learning

Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization

Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization

Offline Reinforcement Learning with Soft Behavior Regularization

Online Policy Optimization for Robust MDP

Offline Reinforcement Learning With Behavior Value Regularization

Robust Offline Reinforcement Learning from Low-Quality Data

Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees

SelfBC: Self Behavior Cloning for Offline Reinforcement Learning