Abstract:Multifingered hand dexterous manipulation is quite challenging in the domain of robotics. One remaining issue is how to achieve compliant behaviors. In this work, we propose a human-in-the-loop learning-control approach for acquiring compliant grasping and manipulation skills of a multifinger robot hand. This approach takes the depth image of the human hand as input and generates the desired force commands for the robot. The markerless vision-based teleoperation system is used for the task demonstration, and an end-to-end neural network model (i.e., TeachNet) is trained to map the pose of the human hand to the joint angles of the robot hand in real-time. To endow the robot hand with compliant human-like behaviors, an adaptive force control strategy is designed to predict the desired force control commands based on the pose difference between the robot hand and the human hand during the demonstration. The force controller is derived from a computational model of the biomimetic control strategy in human motor learning, which allows adapting the control variables (impedance and feedforward force) online during the execution of the reference joint angles. The simultaneous adaptation of the impedance and feedforward profiles enables the robot to interact with the environment compliantly. Our approach has been verified in both simulation and real-world task scenarios based on a multifingered robot hand, that is, the Shadow Hand, and has shown more reliable performances than the current widely used position control mode for obtaining compliant grasping and manipulation behaviors.

Evaluation of state representation methods in robot hand-eye coordination learning from demonstration

InterRep: A Visual Interaction Representation for Robotic Grasping

Reinforcement Learning with Decoupled State Representation for Robot Manipulations

Towards Learning Controllable Representations of Physical Systems

State Representations as Incentives for Reinforcement Learning Agents: A Sim2Real Analysis on Robotic Grasping

State Representation Learning for Task and Motion Planning in Robot Manipulation

A survey of robot learning from demonstration

Diff-LfD: Contact-aware Model-based Learning from Visual Demonstration for Robotic Manipulation Via Differentiable Physics-based Simulation and Rendering.

Learning State Conditioned Linear Mappings for Low-Dimensional Control of Robotic Manipulators

Learning from Successful and Failed Demonstrations via Optimization

A Comparative Study on State-Action Spaces for Learning Viewpoint Selection and Manipulation with Diffusion Policy

Robotic uncalibrated hand-eye coordination based on the extended state observer

Enhancing Visual Reinforcement Learning with State-Action Representation

Learning from Demonstration Framework for Multi-Robot Systems Using Interaction Keypoints and Soft Actor-Critic Methods

Interactive Learning of State Representation through Natural Language Instruction and Explanation

Estimator-Coupled Reinforcement Learning for Robust Purely Tactile In-Hand Manipulation

Learning Task Planning from Multi-Modal Demonstration for Multi-Stage Contact-Rich Manipulation

Human-oriented Representation Learning for Robotic Manipulation

Robot Learning from Demonstration Using Elastic Maps

Image-Based Composite Learning Robot Visual Servoing With an Uncalibrated Eye-to-Hand Camera

Multifingered Robot Hand Compliant Manipulation Based on Vision-Based Demonstration and Adaptive Force Control