Abstract:In-hand object reorientation is necessary for performing many dexterous manipulation tasks, such as tool use in less structured environments that remain beyond the reach of current robots. Prior works built reorientation systems assuming one or many of the following: reorienting only specific objects with simple shapes, limited range of reorientation, slow or quasistatic manipulation, simulation-only results, the need for specialized and costly sensor suites, and other constraints which make the system infeasible for real-world deployment. We present a general object reorientation controller that does not make these assumptions. It uses readings from a single commodity depth camera to dynamically reorient complex and new object shapes by any rotation in real-time, with the median reorientation time being close to seven seconds. The controller is trained using reinforcement learning in simulation and evaluated in the real world on new object shapes not used for training, including the most challenging scenario of reorienting objects held in the air by a downward-facing hand that must counteract gravity during reorientation. Our hardware platform only uses open-source components that cost less than five thousand dollars. Although we demonstrate the ability to overcome assumptions in prior work, there is ample scope for improving absolute performance. For instance, the challenging duck-shaped object not used for training was dropped in 56 percent of the trials. When it was not dropped, our controller reoriented the object within 0.4 radians (23 degrees) 75 percent of the time. Videos are available at: <a class="link-external link-https" href="https://taochenshh.github.io/projects/visual-dexterity" rel="external noopener nofollow">this https URL</a>.

In-Hand Object Rotation via Rapid Motor Adaptation

Rotating without Seeing: Towards In-hand Dexterity through Touch

AnyRotate: Gravity-Invariant In-Hand Object Rotation with Sim-to-Real Touch

Rapid Motor Adaptation for Robotic Manipulator Arms

Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapes

Robotic in-hand manipulation with relaxed optimization

Nonprehensile Manipulation for Rapid Object Spinning via Multisensory Learning from Demonstration

Contact-Implicit Model Predictive Control for Dexterous In-hand Manipulation: A Long-Horizon and Robust Approach

Learning Hierarchical Control for Robust In-Hand Manipulation

Rotating Objects via In-Hand Pivoting using Vision, Force and Touch

Mechanical Intelligence for Prehensile In-Hand Manipulation of Spatial Trajectories.

Planar In-Hand Manipulation Using Primitive Rotations Based on Isometric Transformations

Design and Control of Roller Grasper V2 for In-Hand Manipulation

Adaptive Fingers Coordination for Robust Grasp and In-Hand Manipulation Under Disturbances and Unknown Dynamics

In-Hand Gravitational Pivoting Using Tactile Sensing

Hybrid Learning- and Model-Based Planning and Control of In-Hand Manipulation

TacRot: A Parallel-Jaw Gripper with Rotatable Tactile Sensors for In-Hand Manipulation

Dexterous Soft Hands Linearize Feedback-Control for In-Hand Manipulation

Object-Centric Dexterous Manipulation from Human Motion Data

Estimator-Coupled Reinforcement Learning for Robust Purely Tactile In-Hand Manipulation

General In-Hand Object Rotation with Vision and Touch