Abstract:Although Reinforcement Learning (RL) has demonstrated impressive success in various applications, addressing complex robotic manipulation tasks remains a formidable challenge. Recently, Skill-based approaches that extract reusable skills from offline data and encode them into a latent space are proposed to leverage prior knowledge for accelerating robot learning. However, existing skill learning methods predominantly rely on regularization constraints or reversible mappings to guide skill prior generation, lacking explicit control over the trade-off between exploiting offline knowledge and exploring novel skill behaviors. In this paper, we point out that the challenge of skill exploration lies in the noise within skill embeddings and propose a novel denoising-based skill-based RL framework, DiffSkill. Specifically, our DiffSkill integrates a diffusion-based skill denoiser into the hierarchical architecture, effectively bridging the gap between offline knowledge and learned skill prior embeddings through iterative denoising. Nevertheless, incorporating diffusion models into the skill-based RL framework for robot control faces two main challenges: (i) Uncertain noisy levels of skill embeddings and (ii) Action oscillation during skill transitions. In this regard, we propose a cycle anneal scheduler for dynamic timestep adjustment and an online momentum smoothing strategy to effectively mitigate oscillations during skill transitions, resulting in more stable and superior performance. Extensive comparison experiments across six challenging robotic manipulation tasks demonstrate that DiffSkill consistently outperforms state-of-the-art methods by a significant margin in all downstream tasks. Ablation studies and additional discussions further validate the effectiveness of each component and strategy.

Unsupervised Reinforcement Learning for Transferable Manipulation Skill Discovery

Unsupervised Discovery of Transitional Skills for Deep Reinforcement Learning

Unsupervised Skill Discovery for Robotic Manipulation through Automatic Task Generation

Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning

Learning to combine primitive skills: A step towards versatile robotic manipulation

Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning

Fully Autonomous Real-World Reinforcement Learning with Applications to Mobile Manipulation

Self-Supervised Reinforcement Learning that Transfers using Random Features

Sampling-based Exploration for Reinforcement Learning of Dexterous Manipulation

EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data

Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates

Combining Model-Based $Q$ -Learning with Structural Knowledge Transfer for Robot Skill Learning

Bridging Locomotion and Manipulation Using Reconfigurable Robotic Limbs via Reinforcement Learning

Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost

Efficient Robotic Manipulation Through Offline-to-Online Reinforcement Learning and Goal-Aware State Information

A Reinforcement Learning-Based Framework for Robot Manipulation Skill Acquisition

Task-Oriented Deep Reinforcement Learning for Robotic Skill Acquisition and Control

Diffskill: Improving Reinforcement Learning Through Diffusion-Based Skill Denoiser for Robotic Manipulation

Real-World Robot Reaching Skill Learning Based On Deep Reinforcement Learning

One-shot sim-to-real transfer policy for robotic assembly via reinforcement learning with visual demonstration