Abstract:The International Journal of Robotics Research, Ahead of Print. Learning and imitating behavioral intelligence from human demonstrations is a promising approach towards the intuitive programming of robots for enhanced dynamic dexterity. However, there has been no publicly available dataset in this domain. To address this gap, we introduce the first large-scale dataset and recording framework specifically designed for studying human collaborative dynamic dexterity in throw&catch tasks. The dataset, named H2TC, contains 15,000 multi-view and multi-modal synchronized recordings of diverse Human-Human Throw-and-Catch activities. It involves 34 human subjects with typical motor abilities and a variety of 52 objects frequently manipulated through throw&catch in domestic and/or industrial scenarios. The dataset is supplemented with a hierarchy of manually annotated semantic and dense labels, such as the ground truth human body, hand and object motions captured with specialized high-precision motion tracking systems. These rich annotations make the dataset well-suited for a wide range of robot studies, including both low-level motor skill learning and high-level cognitive planning and recognition. We envision that the proposed dataset and recording framework will facilitate learning pipelines to extract insights on how humans coordinate both intra- and interpersonally to throw and catch objects, ultimately leading to the development of more capable and collaborative robots. The dataset, along with a suite of utility tools, such as those for visualization and annotation, can be accessed from our project page at https://h2tc-roboticsx.github.io/.

Catch It! Learning to Catch in Flight with Mobile Dexterous Hands

DexCatch: Learning to Catch Arbitrary Objects with Dexterous Hands

Dynamic Handover: Throw and Catch with Bimanual Hands

Neural Motion Prediction for In-flight Uneven Object Catching

DexRepNet: Learning Dexterous Robotic Grasping Network with Geometric and Spatial Hand-Object Representations

Modular Neural Network Policies for Learning In-Flight Object Catching with a Robot Hand-Arm System

Advancing robots with greater dynamic dexterity: A large-scale multi-view and multi-modal dataset of human-human throw&catch of arbitrary objects

Run and Catch: Dynamic Object-Catching of Quadrupedal Robots

Object-Centric Dexterous Manipulation from Human Motion Data

Whole-Body Dynamic Throwing with Legged Manipulators

DextrAH-G: Pixels-to-Action Dexterous Arm-Hand Grasping with Geometric Fabrics

TossingBot: Learning to Throw Arbitrary Objects With Residual Physics

Dynamic Object Catching with Quadruped Robot Front Legs

DexTransfer: Real World Multi-fingered Dexterous Grasping with Minimal Human Demonstrations

Playful DoggyBot: Learning Agile and Precise Quadrupedal Locomotion

Improving Soft-Capture Phase Success in Space Debris Removal Missions: Leveraging Deep Reinforcement Learning and Tactile Feedback

Dexterous In-Hand Manipulation of Slender Cylindrical Objects through Deep Reinforcement Learning with Tactile Sensing

Solving Challenging Dexterous Manipulation Tasks With Trajectory Optimisation and Reinforcement Learning

DexH2R: Task-oriented Dexterous Manipulation from Human to Robots

Sampling-based Exploration for Reinforcement Learning of Dexterous Manipulation

Learning dexterous in-hand manipulation