Abstract:With the development of deep learning technology, deep reinforcement learning (DRL) has successfully built intelligent agents in sequential decision-making problems through interaction with image-based environments. However, learning from unlimited interaction is impractical and sample inefficient because training an agent requires many trial and error and numerous samples. One response to this problem is sample-efficient DRL, a research area that encourages learning effective state representations in limited interactions with image-based environments. Previous methods could effectively surpass human performance by training an RL agent using self-supervised learning and data augmentation to learn good state representations from a given interaction. However, most of the existing methods only consider similarity of image observations so that they are hard to capture semantic representations. To address these challenges, we propose spatio-temporal and action-based contrastive representation (STACoRe) learning for sample-efficient DRL. STACoRe performs two contrastive learning to learn proper state representations. One uses the agent's actions as pseudo labels, and the other uses spatio-temporal information. In particular, when performing the action-based contrastive learning, we propose a method that automatically selects data augmentation techniques suitable for each environment for stable model training. We train the model by simultaneously optimizing an action-based contrastive loss function and spatio-temporal contrastive loss functions in an end-to-end manner. This leads to improving sample efficiency for DRL. We use 26 benchmark games in Atari 2600 whose environment interaction is limited to only 100k steps. The experimental results confirm that our method is more sample efficient than existing methods. The code is available at https://github.com/dudwojae/STACoRe.

Deep Reinforcement Learning With Macro-Actions

Time‐in‐action RL

Macro-Action-Based Deep Multi-Agent Reinforcement Learning

Multi-Agent/Robot Deep Reinforcement Learning with Macro-Actions (Student Abstract)

An Alternative Curriculum Learning Approach with Macro Actions for Deep Reinforcement Learning

Macro-Action-Based Multi-Agent/Robot Deep Reinforcement Learning under Partial Observability

Playing Atari with Deep Reinforcement Learning

Model-Based Reinforcement Learning for Atari

Data Efficient Deep Reinforcement Learning with Action-Ranked Temporal Difference Learning

Macro Action Reinforcement Learning with Sequence Disentanglement using Variational Autoencoder

Learn to Interpret Atari Agents.

Hierarchical Meta-Reinforcement Learning via Automated Macro-Action Discovery

State of the Art Control of Atari Games Using Shallow Reinforcement Learning

At Human Speed: Deep Reinforcement Learning with Action Delay

STACoRe: Spatio-temporal and Action-Based Contrastive Representations for Reinforcement Learning in Atari

Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation

HiMacMic: Hierarchical Multi-Agent Deep Reinforcement Learning with Dynamic Asynchronous Macro Strategy

Generalizing soft actor-critic algorithms to discrete action spaces

Symbolic Task Inference in Deep Reinforcement Learning

Learning Macromanagement in Starcraft by Deep Reinforcement Learning