Abstract:Offline reinforcement learning (RL) learns policies entirely from static datasets, thereby avoiding the challenges associated with online data collection. Practical applications of offline RL will inevitably require learning from datasets where the variability of demonstrated behaviors changes non-uniformly across the state space. For example, at a red light, nearly all human drivers behave similarly by stopping, but when merging onto a highway, some drivers merge quickly, efficiently, and safely, while many hesitate or merge dangerously. Both theoretically and empirically, we show that typical offline RL methods, which are based on distribution constraints fail to learn from data with such non-uniform variability, due to the requirement to stay close to the behavior policy to the same extent across the state space. Ideally, the learned policy should be free to choose per state how closely to follow the behavior policy to maximize long-term return, as long as the learned policy stays within the support of the behavior policy. To instantiate this principle, we reweight the data distribution in conservative Q-learning (CQL) to obtain an approximate support constraint formulation. The reweighted distribution is a mixture of the current policy and an additional policy trained to mine poor actions that are likely under the behavior policy. Our method, CQL (ReDS), is simple, theoretically motivated, and improves performance across a wide range of offline RL problems in Atari games, navigation, and pixel-based manipulation.

Offline Reinforcement Learning at Multiple Frequencies

Offline Reinforcement Learning for Wireless Network Optimization with Mixture Datasets

Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints

Offline Reinforcement Learning from Datasets with Structured Non-Stationarity

Offline Decentralized Multi-Agent Reinforcement Learning

Leveraging Offline Data in Online Reinforcement Learning

Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning

Equivariant Offline Reinforcement Learning

Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions

Adaptive Policy Learning for Offline-to-Online Reinforcement Learning

Simple Ingredients for Offline Reinforcement Learning

Robust Reinforcement Learning using Offline Data

Adaptable Conservative Q-Learning for Offline Reinforcement Learning.

A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems

Improving Offline Reinforcement Learning with Inaccurate Simulators

Planning, Fast and Slow: Online Reinforcement Learning with Action-Free Offline Data Via Multiscale Planners

Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices

Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage

Online Tuning for Offline Decentralized Multi-Agent Reinforcement Learning

Mildly Conservative Q-Learning for Offline Reinforcement Learning

A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning