Abstract:Although Reinforcement Learning (RL) has demonstrated impressive success in various applications, addressing complex robotic manipulation tasks remains a formidable challenge. Recently, Skill-based approaches that extract reusable skills from offline data and encode them into a latent space are proposed to leverage prior knowledge for accelerating robot learning. However, existing skill learning methods predominantly rely on regularization constraints or reversible mappings to guide skill prior generation, lacking explicit control over the trade-off between exploiting offline knowledge and exploring novel skill behaviors. In this paper, we point out that the challenge of skill exploration lies in the noise within skill embeddings and propose a novel denoising-based skill-based RL framework, DiffSkill. Specifically, our DiffSkill integrates a diffusion-based skill denoiser into the hierarchical architecture, effectively bridging the gap between offline knowledge and learned skill prior embeddings through iterative denoising. Nevertheless, incorporating diffusion models into the skill-based RL framework for robot control faces two main challenges: (i) Uncertain noisy levels of skill embeddings and (ii) Action oscillation during skill transitions. In this regard, we propose a cycle anneal scheduler for dynamic timestep adjustment and an online momentum smoothing strategy to effectively mitigate oscillations during skill transitions, resulting in more stable and superior performance. Extensive comparison experiments across six challenging robotic manipulation tasks demonstrate that DiffSkill consistently outperforms state-of-the-art methods by a significant margin in all downstream tasks. Ablation studies and additional discussions further validate the effectiveness of each component and strategy.

Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback

GSC: A Graph-Based Skill Composition Framework for Robot Learning

EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data

DexSkills: Skill Segmentation Using Haptic Data for Learning Autonomous Long-Horizon Robotic Manipulation Tasks

SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deployment

Unsupervised Skill Discovery for Robotic Manipulation through Automatic Task Generation

Diffskill: Improving Reinforcement Learning Through Diffusion-Based Skill Denoiser for Robotic Manipulation

SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions

Efficient Robot Skill Learning with Imitation from a Single Video for Contact-Rich Fabric Manipulation

Accelerating Reinforcement Learning with Learned Skill Priors

SLIM: Skill Learning with Multiple Critics

A data-efficient goal-directed deep reinforcement learning method for robot visuomotor skill

Skill-Critic: Refining Learned Skills for Hierarchical Reinforcement Learning

Learning a Universal Human Prior for Dexterous Manipulation from Human Preference

DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools

Learning and Retrieval from Prior Data for Skill-based Imitation Learning

SKID RAW: Skill Discovery from Raw Trajectories

Skills Made to Order: Efficient Acquisition of Robot Cooking Skills Guided by Multiple Forms of Internet Data

Practice Makes Perfect: Planning to Learn Skill Parameter Policies

Offline Imitation Learning Through Graph Search and Retrieval

RSG: Fast Learning Adaptive Skills for Quadruped Robots by Skill Graph