Grasping Diverse Objects with Simulated Humanoids

Zhengyi Luo,Jinkun Cao,Sammy Christen,Alexander Winkler,Kris Kitani,Weipeng Xu

2024-07-16

Abstract:We present a method for controlling a simulated humanoid to grasp an object and move it to follow an object trajectory. Due to the challenges in controlling a humanoid with dexterous hands, prior methods often use a disembodied hand and only consider vertical lifts or short trajectories. This limited scope hampers their applicability for object manipulation required for animation and simulation. To close this gap, we learn a controller that can pick up a large number (>1200) of objects and carry them to follow randomly generated trajectories. Our key insight is to leverage a humanoid motion representation that provides human-like motor skills and significantly speeds up training. Using only simplistic reward, state, and object representations, our method shows favorable scalability on diverse object and trajectories. For training, we do not need dataset of paired full-body motion and object trajectories. At test time, we only require the object mesh and desired trajectories for grasping and transporting. To demonstrate the capabilities of our method, we show state-of-the-art success rates in following object trajectories and generalizing to unseen objects. Code and models will be released.

Robotics,Graphics

What problem does this paper attempt to address?

This paper presents a method for controlling a simulated humanoid robot equipped with two dexterous hands to grasp objects and move them along predefined trajectories. Due to the challenges of manipulating objects with complex hand movements for humanoid robots, previous methods often used hand models without a body and considered only vertical lifting or short-distance trajectories. This approach limited their applications in object manipulation in animation and simulation.

To address this problem, the paper introduces a controller called "Omnigrasp" that can learn to grasp a large number (over 1200) of objects with different shapes and transport them along randomly generated trajectories. The key innovation is to utilize a pre-trained general-purpose dexterous motion representation to provide human-like motion skills, significantly accelerating the training process. This approach only uses simple rewards, states, and object representations, demonstrating good scalability on various objects and trajectories. No paired data of full-body motion and object trajectories are required during training. During testing, only the object's mesh and the desired grasping and transportation trajectories are needed.

The paper demonstrates the success rate of this approach in following object trajectories and its generalization ability to unseen objects, and the code and models will be released in the future. Furthermore, experiments show that Omnigrasp achieves state-of-the-art success rates in executing complex object trajectory tracking tasks and is capable of handling unseen objects.