Abstract:Discrete-continuous hybrid action space is a natural setting in many practical problems, such as robot control and game AI. However, most previous Reinforcement Learning (RL) works only demonstrate the success in controlling with either discrete or continuous action space, while seldom take into account the hybrid action space. One naive way to address hybrid action RL is to convert the hybrid action space into a unified homogeneous action space by discretization or continualization, so that conventional RL algorithms can be applied. However, this ignores the underlying structure of hybrid action space and also induces the scalability issue and additional approximation difficulties, thus leading to degenerated results. In this paper, we propose Hybrid Action Representation (HyAR) to learn a compact and decodable latent representation space for the original hybrid action space. HyAR constructs the latent space and embeds the dependence between discrete action and continuous parameter via an embedding table and conditional Variantional Auto-Encoder (VAE). To further improve the effectiveness, the action representation is trained to be semantically smooth through unsupervised environmental dynamics prediction. Finally, the agent then learns its policy with conventional DRL algorithms in the learned representation space and interacts with the environment by decoding the hybrid action embeddings to the original action space. We evaluate HyAR in a variety of environments with discrete-continuous action space. The results demonstrate the superiority of HyAR when compared with previous baselines, especially for high-dimensional action spaces.

Macro Action Reinforcement Learning with Sequence Disentanglement using Variational Autoencoder

Macro-Action-Based Multi-Agent/Robot Deep Reinforcement Learning under Partial Observability

Macro-Action-Based Deep Multi-Agent Reinforcement Learning

Deep Reinforcement Learning With Macro-Actions

Multi-Agent/Robot Deep Reinforcement Learning with Macro-Actions (Student Abstract)

Hierarchical Meta-Reinforcement Learning via Automated Macro-Action Discovery

REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processes

An Alternative Curriculum Learning Approach with Macro Actions for Deep Reinforcement Learning

DCT: Dual Channel Training of Action Embeddings for Reinforcement Learning with Large Discrete Action Spaces

Towards Governing Agent's Efficacy: Action-Conditional $β$-VAE for Deep Transparent Reinforcement Learning

Learning Action Representations for Reinforcement Learning

HiMacMic: Hierarchical Multi-Agent Deep Reinforcement Learning with Dynamic Asynchronous Macro Strategy

Learning Disentangled Representations for Recommendation

HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation

In-Context Reinforcement Learning for Variable Action Spaces

Exact Reduction of Huge Action Spaces in General Reinforcement Learning

A Compression-Inspired Framework for Macro Discovery

Virtual Action Actor-Critic Framework for Exploration (Student Abstract)

Effective Multi-Agent Deep Reinforcement Learning Control with Relative Entropy Regularization

Action Mapping for Reinforcement Learning in Continuous Environments with Constraints

UAV Swarm Confrontation Using Hierarchical Multiagent Reinforcement Learning