Abstract:Reinforcement learning (RL) has excelled in sequential decision-making and control tasks, yet traditional RL algorithms are limited by adherence to a single control style in identical scenarios, failing to address varied control preferences. Existing multi-style RL methods typically require customized reward or objective functions tailored to specific control styles, which may not be feasible when diverse driving styles are necessary. To overcome these limitations, we propose the multi-style distributional soft actor-critic (M-DSAC) algorithm, capable of learning a single policy that supports multiple control behaviors. We begin by developing a multi-style policy iteration (MPI) framework that learns the entire distribution of returns, known as the value distribution, rather than just focusing on the expected return (i.e., the $Q$ value). In this framework, we utilize the quantile index of the value distribution as a style indicator, enhancing the inputs of both the policy and its corresponding value distribution with these quantile indices. Building upon the MPI framework, the M-DSAC algorithm employs a parameterized diagonal Gaussian function to approximate the value distribution. This approach enables efficient computation of different value quantiles by combining the value distribution's mean and standard deviations with appropriate coefficients. By optimizing the policy across different quantiles, M-DSAC efficiently learns a versatile policy that can handle a range of control styles without the burden of significant computing costs. Experimental evaluations using MuJoCo benchmarks and real-world robot control tasks confirm the effectiveness of M-DSAC, showcasing its broad practical applicability.

Revisiting Gaussian mixture critics in off-policy reinforcement learning: a sample-based approach

Actor-critic algorithm based on Gaussian process

Modeling-Learning-Based Actor-Critic Algorithm with Gaussian Process Approximator

Generative Actor-Critic: An Off-policy Algorithm Using the Push-forward Model

Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

Gaussian Process Policy Optimization

ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages

A Novel Heterogeneous Actor-critic Algorithm with Recent Emphasizing Replay Memory

Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors

Off-Policy Neural Fitted Actor-Critic

The Actor-Dueling-Critic Method for Reinforcement Learning.

Doubly Robust Off-Policy Actor-Critic Algorithms for Reinforcement Learning

CGAR: Critic Guided Action Redistribution in Reinforcement Leaning

A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning

OPAC: Opportunistic Actor-Critic

Actor-Critic Algorithms With Epsilon-Greedy Gaussian Policy In Multidimensional Continuous Action Spaces

Multi-agent Gradient-Based Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning

Multi-Style Distributional Soft Actor-Critic: Learning a Unified Policy for Diverse Control Behaviors

The Reactor: A fast and sample-efficient Actor-Critic agent for Reinforcement Learning

Neural Network Compatible Off-Policy Natural Actor-Critic Algorithm

Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments