Abstract:Learning from previously collected data via behavioral cloning or offline reinforcement learning (RL) is a powerful recipe for scaling generalist agents by avoiding the need for expensive online learning. Despite strong generalization in some respects, agents are often remarkably brittle to minor visual variations in control-irrelevant factors such as the background or camera viewpoint. In this paper, we present theDeepMind Control Visual Benchmark (DMC-VB), a dataset collected in the DeepMind Control Suite to evaluate the robustness of offline RL agents for solving continuous control tasks from visual input in the presence of visual distractors. In contrast to prior works, our dataset (a) combines locomotion and navigation tasks of varying difficulties, (b) includes static and dynamic visual variations, (c) considers data generated by policies with different skill levels, (d) systematically returns pairs of state and pixel observation, (e) is an order of magnitude larger, and (f) includes tasks with hidden goals. Accompanying our dataset, we propose three benchmarks to evaluate representation learning methods for pretraining, and carry out experiments on several recently proposed methods. First, we find that pretrained representations do not help policy learning on DMC-VB, and we highlight a large representation gap between policies learned on pixel observations and on states. Second, we demonstrate when expert data is limited, policy learning can benefit from representations pretrained on (a) suboptimal data, and (b) tasks with stochastic hidden goals. Our dataset and benchmark code to train and evaluate agents are available at: <a class="link-external link-https" href="https://github.com/google-deepmind/dmc_vision_benchmark" rel="external noopener nofollow">this https URL</a>.

CURL: Contrastive Unsupervised Representations for Reinforcement Learning

Masked Contrastive Representation Learning for Reinforcement Learning

CURLing the Dream: Contrastive Representations for World Modeling in Reinforcement Learning

M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulation

Contrastive Unsupervised Representation Learning With Optimize-Selected Training Samples

MetaCURL: Non-stationary Concave Utility Reinforcement Learning

Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentation

Decoupling Representation Learning from Reinforcement Learning

Sequential and Dynamic Constraint Contrastive Learning for Reinforcement Learning.

DMC-VB: A Benchmark for Representation Learning for Control with Visual Distractors

CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery

Unsupervised Control Through Non-Parametric Discriminative Rewards

Efficient Model-Based Concave Utility Reinforcement Learning through Greedy Mirror Descent

Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learning

Visual Pretraining via Contrastive Predictive Model for Pixel-Based Reinforcement Learning

Towards Control-Centric Representations in Reinforcement Learning from Images

Depth-CUPRL: Depth-Imaged Contrastive Unsupervised Prioritized Representations in Reinforcement Learning for Mapless Navigation of Unmanned Aerial Vehicles

Curious Representation Learning for Embodied Intelligence

MOOSS: Mask-Enhanced Temporal Contrastive Learning for Smooth State Evolution in Visual Reinforcement Learning

An adaptive deep reinforcement learning framework enables curling robots with human-like performance in real-world conditions