Abstract:Complex and contact-rich robotic manipulation tasks, particularly those that involve multi-fingered hands and underactuated object manipulation, present a significant challenge to any control method. Methods based on reinforcement learning offer an appealing choice for such settings, as they can enable robots to learn to delicately balance contact forces and dexterously reposition objects without strong modeling assumptions. However, running reinforcement learning on real-world dexterous manipulation systems often requires significant manual engineering. This negates the benefits of autonomous data collection and ease of use that reinforcement learning should in principle provide. In this paper, we describe a system for vision-based dexterous manipulation that provides a "programming-free" approach for users to define new tasks and enable robots with complex multi-fingered hands to learn to perform them through interaction. The core principle underlying our system is that, in a vision-based setting, users should be able to provide high-level intermediate supervision that circumvents challenges in teleoperation or kinesthetic teaching which allow a robot to not only learn a task efficiently but also to autonomously practice. Our system includes a framework for users to define a final task and intermediate sub-tasks with image examples, a reinforcement learning procedure that learns the task autonomously without interventions, and experimental results with a four-finger robotic hand learning multi-stage object manipulation tasks directly in the real world, without simulation, manual modeling, or reward engineering.

DexMV: Imitation Learning for Dexterous Manipulation from Human Videos

Learning Robot Manipulation Skills from Human Demonstration Videos Using Two-Stream 2-D/3-D Residual Networks with Self-Attention

DexRepNet: Learning Dexterous Robotic Grasping Network with Geometric and Spatial Hand-Object Representations

DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning

From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation From Single-Camera Teleoperation

DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video

DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation

DexH2R: Task-oriented Dexterous Manipulation from Human to Robots

ViViDex: Learning Vision-based Dexterous Manipulation from Human Videos

Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation

DexDeform: Dexterous Deformable Object Manipulation with Human Demonstrations and Differentiable Physics

H-InDex: Visual Reinforcement Learning with Hand-Informed Representations for Dexterous Manipulation

Learning Generalizable Dexterous Manipulation from Human Grasp Affordance

RealDex: Towards Human-like Grasping for Robotic Dexterous Hand

Object-Centric Dexterous Manipulation from Human Motion Data

Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidance

Learning Diverse Bimanual Dexterous Manipulation Skills from Human Demonstrations

Giving Robots a Hand: Learning Generalizable Manipulation with Eye-in-Hand Human Video Demonstrations

Watch and Act: Learning Robotic Manipulation from Visual Demonstration.

Holo-Dex: Teaching Dexterity with Immersive Mixed Reality

DexSkills: Skill Segmentation Using Haptic Data for Learning Autonomous Long-Horizon Robotic Manipulation Tasks