Abstract:In meta reinforcement learning (meta RL), an agent seeks a Bayes-optimal policy -- the optimal policy when facing an unknown task that is sampled from some known task distribution. Previous approaches tackled this problem by inferring a belief over task parameters, using variational inference methods. Motivated by recent successes of contrastive learning approaches in RL, such as contrastive predictive coding (CPC), we investigate whether contrastive methods can be used for learning Bayes-optimal behavior. We begin by proving that representations learned by CPC are indeed sufficient for Bayes optimality. Based on this observation, we propose a simple meta RL algorithm that uses CPC in lieu of variational belief inference. Our method, ContraBAR, achieves comparable performance to state-of-the-art in domains with state-based observation and circumvents the computational toll of future observation reconstruction, enabling learning in domains with image-based observations. It can also be combined with image augmentations for domain randomization and used seamlessly in both online and offline meta RL settings.

What problem does this paper attempt to address?

The paper attempts to address the problem of how to effectively learn a Bayes-optimal policy in meta reinforcement learning (meta RL). Specifically, the paper explores whether contrastive learning methods can be used to learn Bayes-optimal behavior, especially when dealing with tasks involving image observations. Traditional variational inference methods face challenges such as high computational cost and optimization difficulties when handling image data, whereas contrastive learning methods like Contrastive Predictive Coding (CPC) have shown strong representation learning capabilities in other domains. Therefore, the paper proposes a meta reinforcement learning algorithm based on contrastive learning—ContraBAR, aiming to overcome the limitations of existing methods and achieve good performance in both state observation and image observation tasks. The main contributions of the paper include: 1. Demonstrating that the representations learned using CPC are indeed sufficient statistics required for control and can serve as inputs for a Bayes-optimal policy. 2. Proposing a simple meta reinforcement learning algorithm, ContraBAR, which uses CPC instead of variational belief inference, avoiding the computational burden of future observation reconstruction, thus enabling learning in the image observation domain. 3. Experimentally validating that ContraBAR performs comparably to current state-of-the-art methods in state observation tasks and significantly outperforms variational methods in image observation tasks. Additionally, it can seamlessly integrate with image augmentation and domain randomization, making it suitable for both online and offline meta reinforcement learning settings. In summary, the goal of the paper is to provide a new, efficient, and general approach to meta reinforcement learning by introducing contrastive learning methods.

ContraBAR: Contrastive Bayes-Adaptive Deep RL

Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learning

Deep Interactive Bayesian Reinforcement Learning via Meta-Learning

VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning

Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning

Contrastive Example-Based Control

Combining Reconstruction and Contrastive Methods for Multimodal Representations in RL

REBAR: Retrieval-Based Reconstruction for Time-series Contrastive Learning

Contrastive Preference Learning: Learning from Human Feedback without RL

Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator

All by Myself: Learning Individualized Competitive Behaviour with a Contrastive Reinforcement Learning optimization

Action-driven contrastive representation for reinforcement learning

Contrastive Difference Predictive Coding

Masked Contrastive Representation Learning for Reinforcement Learning

Latent Context Based Soft Actor-Critic

RLBOF: Reinforcement Learning from Bayesian Optimization Feedback

Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization

Data-Efficient Task Generalization via Probabilistic Model-based Meta Reinforcement Learning

Sequential and Dynamic Constraint Contrastive Learning for Reinforcement Learning.

Decoupling Representation Learning from Reinforcement Learning