Abstract:Employing cognitive robots, capable of throwing and catching, is a strategy aimed at expediting the logistics process within Industry 4.0's smart manufacturing plants, specifically for the transportation of small-sized manufacturing parts. Since the flight of mechanically thrown objects is inherently unpredictable, it is crucial for the catching robot to observe the initial trajectory with utmost precision and intelligently forecast the final catching position to ensure accurate real-time grasping. This study utilizes multi-camera tracking to monitor mechanically thrown objects. It involves the creation of a 3D simulation that facilitates controlled mechanical throwing of objects within the internal logistics environment of Industry 4.0. The developed simulation empowers users to define the attributes of the thrown object and capture its trajectory using a simulated pinhole camera, which can be positioned at any desired location and orientation within the in-plant logistics environment of flexible manufacturing systems. The simulation facilitated ample experimentation to be conducted for determining the optimal camera positions for accurately observing the 3D interception positions of a flying object based on its apparent size on the camera's sensor plane. Subsequently, a variety of calibrated multi-camera setups were experimented while placing cameras at identified optimal positions. Based on the obtained results, the most effective multi-camera configuration setup is derived. Finally, a training dataset is prepared for 3000 simulated throwing experiments where the initial part of the trajectory consists of observed interception positions, through derived best multi-camera setup, and the final part consists of actual positions. The encoder–decoder Bi-LSTM deep neural network is proposed and trained on this dataset. The trained model outperformed the current state-of-the-art by accurately predicting the final 3D catching point, achieving a mean average error of 5 mm and a root-mean-square error of 7 mm in 200 real-world test experiments.

Advancing robots with greater dynamic dexterity: A large-scale multi-view and multi-modal dataset of human-human throw&catch of arbitrary objects

Dynamic Handover: Throw and Catch with Bimanual Hands

DA$^{2}$ Dataset: Toward Dexterity-Aware Dual-Arm Grasping.

RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot

Catch It! Learning to Catch in Flight with Mobile Dexterous Hands

Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation

DexCatch: Learning to Catch Arbitrary Objects with Dexterous Hands

RealDex: Towards Human-like Grasping for Robotic Dexterous Hand

DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation

Multi-camera tracking of mechanically thrown objects for automated in-plant logistics by cognitive robots in Industry 4.0

Learning Human-to-Robot Dexterous Handovers for Anthropomorphic Hand

Learn to Predict How Humans Manipulate Large-Sized Objects From Interactive Motions

BEHAVE: Dataset and Method for Tracking Human Object Interactions

TossingBot: Learning to Throw Arbitrary Objects With Residual Physics

H2O: A Benchmark for Visual Human-human Object Handover Analysis

DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

DeepClaw 2.0: A Data Collection Platform for Learning Human Manipulation

Multi-task real-robot data with gaze attention for dual-arm fine manipulation

MultiGripperGrasp: A Dataset for Robotic Grasping from Parallel Jaw Grippers to Dexterous Hands

AIR-Act2Act: Human-human interaction dataset for teaching non-verbal social behaviors to robots

HabitatDyn Dataset: Dynamic Object Detection to Kinematics Estimation