Abstract:A novel concept of vision-based intelligent control of robotic arms is developed here in this work. This work enables the controlling of robotic arms motion only with visual inputs, that is, controlling by showing the videos of correct movements. This work can broadly be sub-divided into two segments. The first part of this work is to develop an unsupervised vision-based method to control robotic arm in 2-D plane, and the second one is with deep CNN in the same task in 3-D plane. The first method is unsupervised, where our aim is to perform mimicking of human arm motion in real-time by a manipulator. We developed a network, namely the vision-to-motion optical network (DON), where the input should be a video stream containing hand movements of human, the the output would be out the velocity and torque information of the hand movements shown in the videos. The output information of the DON is then fed to the robotic arm by enabling it to generate motion according to the real hand videos. The method has been tested with both live-stream video feed as well as on recorded video obtained from a monocular camera even by intelligently predicting the trajectory of human hand hand when it gets occluded. This is why the mimicry of the arm incorporates some intelligence to it and becomes intelligent mimic (i-mimic). Alongside the unsupervised method another method has also been developed deploying the deep neural network technique with CNN (Convolutional Neural Network) to perform the mimicking, where labelled datasets are used for training. The same dataset, as used in the unsupervised DON-based method, is used in the deep CNN method, after manual annotations. Both the proposed methods are validated with off-line as well as with on-line video datasets in real-time. The entire methodology is validated with real-time 1-link and simulated n-link manipulators alongwith suitable comparisons.

Copycat hand — robot hand imitating human motions at high speed and with high accuracy

Entertainment Robot Hand Gesture Recognition

Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube

Soft and Rigid Object Grasping With Cross-Structure Hand Using Bilateral Control-Based Imitation Learning

Image-recognition-based System for Precise Hand Function Evaluation

SmartHand: Towards Embedded Smart Hands for Prosthetic and Robotic Applications

The Manipulation of Real-Time Kinect-Based Robotic Arm Using Double-Hand Gestures

Eye-in-Hand Robotic Arm Gripping System Based on Machine Learning and State Delay Optimization

Robot to Human Object Handover using Vision and Joint Torque Sensor Modalities

Efficient Hand Gesture Recognition for Human-Robot Interaction

Towards Predicting Fine Finger Motions from Ultrasound Images via Kinematic Representation

Kinematic Motion Retargeting for Contact-Rich Anthropomorphic Manipulations

Leveraging Pretrained Latent Representations for Few-Shot Imitation Learning on a Dexterous Robotic Hand

Improved mKLT and low layered HG-CNN based dynamic gesture recognition hardware system

FastHand: Fast monocular hand pose estimation on embedded systems

Development of a prosthetic hand control system Based on general object recognition analysis of recognition accuracy during approach phase

Motion Similarity Evaluation between Human and a Tri-Co Robot during Real-Time Imitation with a Trajectory Dynamic Time Warping Model

Real time robotic arm control using hand gestures

A Novel Multicamera System for High-Speed Touchless Palm Recognition

HandO: a Hybrid 3D Hand–object Reconstruction Model for Unknown Objects

Controlling by Showing: i-Mimic: A Video-based Method to Control Robotic Arms