Abstract:Soft actor-critic (SAC) in reinforcement learning is expected to be one of the next-generation robot control schemes. Its ability to maximize policy entropy would make a robotic controller robust to noise and perturbation, which is useful for real-world robot applications. However, the priority of maximizing the policy entropy is automatically tuned in the current implementation, the rule of which can be interpreted as one for equality constraint, binding the policy entropy into its specified lower bound. The current SAC is therefore no longer maximize the policy entropy, contrary to our expectation. To resolve this issue in SAC, this paper improves its implementation with a learnable state-dependent slack variable for appropriately handling the inequality constraint to maximize the policy entropy by reformulating it as the corresponding equality constraint. The introduced slack variable is optimized by a switching-type loss function that takes into account the dual objectives of satisfying the equality constraint and checking the lower bound. In Mujoco and Pybullet simulators, the modified SAC statistically achieved the higher robustness for adversarial attacks than before while regularizing the norm of action. A real-robot variable impedance task was demonstrated for showing the applicability of the modified SAC to real-world robot control. In particular, the modified SAC maintained adaptive behaviors for physical human-robot interaction, which had no experience at all during training. <a class="link-external link-https" href="https://youtu.be/EH3xVtlVaJw" rel="external noopener nofollow">this https URL</a>

Soft Actor-Critic with Inhibitory Networks for Faster Retraining

Self-play Reinforcement Learning with Comprehensive Critic in Computer Games

PAC-Bayesian Soft Actor-Critic Learning

Latent Context Based Soft Actor-Critic

Context-Based Soft Actor Critic for Environments with Non-stationary Dynamics

A Graph-Based Soft Actor Critic Approach in Multi-Agent Reinforcement Learning

Bayesian Soft Actor-Critic: A Directed Acyclic Strategy Graph Based Deep Reinforcement Learning

Revisiting Discrete Soft Actor-Critic

Generalizing soft actor-critic algorithms to discrete action spaces

Bayesian Strategy Networks Based Soft Actor-Critic Learning

OPAC: Opportunistic Actor-Critic

CAT-SAC: Soft Actor-Critic with Curiosity-Aware Entropy Temperature

A Strategy-Oriented Bayesian Soft Actor-Critic Model

Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past

Decomposed Soft Actor-Critic Method for Cooperative Multi-Agent Reinforcement Learning

Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience

Soft Actor-Critic Algorithm with Truly-satisfied Inequality Constraint

Averaged Soft Actor-Critic for Deep Reinforcement Learning.

An improved Soft Actor-Critic strategy for optimal energy management

Selector-Actor-Critic and Tuner-Actor-Critic Algorithms for Reinforcement Learning

Robot Skill Adaptation via Soft Actor-Critic Gaussian Mixture Models