Real-Time Robotic Hand Control Using Hand Gestures

Jagdish Lal Raheja,Radhey Shyam,G. Arun Rajsekhar,P. Bhanu Prasad,Jagdish Lal,G. Arun,P. Bhanu

DOI: https://doi.org/10.5772/25512

2012-02-03

Abstract:Real-Time Robotic Hand Control Using Hand Gestures | InTechOpen, Published on: 2012-02-03. Authors: Jagdish Lal Raheja, Radhey Shyam, G. Arun Rajsekhar, et

What problem does this paper attempt to address?

Real time robotic arm control using hand gestures

Ganesh B Choudhary,Ram B V Chethan,Ganesh Choudhary B,Chethan Ram B V

DOI: https://doi.org/10.1109/ichpca.2014.7045349

2014-12-01

Abstract:In this paper we propose a way to accomplish Human Computer Interface absolutely in electronic way (without mechanical sensors). The idea is to extirpate old techniques of controlling Robotic arm using joysticks, buttons and supersede with more intuitive technique ie., to control robotic arm by hand motion or gesture. Here we propound an approach to achieve the aforementioned idea employing Image processing technique using web camera. We detect the vital features of hand: fingers by computational geometry calculation enabling real time interaction between hand gestures and Robot. Our system can meticulously locate fingers even when fore-arm is involved. And the system can sustain a certain rotation of palm and fore-arm, which augments the freedom of use in palm center estimation.
Real Time Robotic Arm using Leap Motion Controller

R Mariappan,P Gayathri,P Pushpalatha,V Sri Sidhvi Rishik,T. Satish

DOI: https://doi.org/10.1088/1742-6596/2466/1/012023

2023-03-01

Journal of Physics: Conference Series

Abstract:Abstract Robots are playing key role in today’s world. Robots are actively taking part in the industries, hospitals, defense applications as well. This paper describes about how hand gesture movements can control the robotic arm. Hand gesture data is recorded by the leap motion controller. That captured gestures are then processed and sent to the microcontroller. In this paper we are intended to design a robotic arm that is completely useful to the people with disabilities. We are using new kind of sensor called Leap motion controller, which captures the hand gestures in 3D. we develop a robotic arm that is useful for the paralyzed people, industrial applications, medical fields by just simply using the hands, we can guide the leap motion controller so that the robotic arm can perform its operations.
The Manipulation of Real-Time Kinect-Based Robotic Arm Using Double-Hand Gestures

Chern-Sheng Lin,Pei-Chi Chen,Yu-Ching Pan,Che-Ming Chang,Kuo-Liang Huang

DOI: https://doi.org/10.1155/2020/9270829

IF: 2.336

2020-01-25

Journal of Sensors

Abstract:This study focused on utilizing the Kinect depth sensor to track double-hand gestures and control a real-time robotic arm. The control system is mainly composed of the microprocessor, a color camera, the depth sensor, and the robotic arm. The Kinect depth sensor was used to take photos of the human body to analyze the skeleton of a human body and obtain the relevant information. Such information was used to identify the gestures of the left hand and the left palm of the user. The gesture of left hand was used as an input command device. The gesture of the right hand was used for imitation movement teaching of robotic arm. From the depth sensor, the real-time images of the human body and the deep information of each joint were collected and converted to the relative positions of the robotic arm. Combining forward kinematics and inverse kinematics and D-H link, the gesture information of the right hand was calculated, which was converted via coordinates into each angle of the motor of the robotic arm. From the color camera, when the left palm was not detected, the user could simply use the right hand to control the action and movement of the real-time robotic arm. When the left palm was detected and 5 fingertips were identified, it meant the start of recording the real-time imitation movement of the robotic arm by the right hand. When 0 fingertip was identified, it meant the stoppage of the above recording. When 2 fingertips were identified, the user could not only control the real-time robotic arm but also repeat the recorded actions.

engineering, electrical & electronic,instruments & instrumentation
Controlling a remotely located Robot using Hand Gestures in real time: A DSP implementation

J L Raheja,G A Rajsekhar,Ankit Chaudhary

DOI: https://doi.org/10.48550/arXiv.1707.07672

IF: 6.4588

2017-07-23

Human-Computer Interaction

Abstract:Telepresence is a necessity for present time as we can't reach everywhere and also it is useful in saving human life at dangerous places. A robot, which could be controlled from a distant location, can solve these problems. This could be via communication waves or networking methods. Also controlling should be in real time and smooth so that it can actuate on every minor signal in an effective way. This paper discusses a method to control a robot over the network from a distant location. The robot was controlled by hand gestures which were captured by the live camera. A DSP board TMS320DM642EVM was used to implement image pre-processing and fastening the whole system. PCA was used for gesture classification and robot actuation was done according to predefined procedures. Classification information was sent over the network in the experiment. This method is robust and could be used to control any kind of robot over distance.
Hands-On Robotics: Enabling Communication Through Direct Gesture Control

Max Pascher,Alia Saad,Jonathan Liebers,Roman Heger,Jens Gerken,Stefan Schneegass,Uwe Gruene

DOI: https://doi.org/10.1145/3610978.3640635

2024-01-17

Abstract:Effective Human-Robot Interaction (HRI) is fundamental to seamlessly integrating robotic systems into our daily lives. However, current communication modes require additional technological interfaces, which can be cumbersome and indirect. This paper presents a novel approach, using direct motion-based communication by moving a robot's end effector. Our strategy enables users to communicate with a robot by using four distinct gestures -- two handshakes ('formal' and 'informal') and two letters ('W' and 'S'). As a proof-of-concept, we conducted a user study with 16 participants, capturing subjective experience ratings and objective data for training machine learning classifiers. Our findings show that the four different gestures performed by moving the robot's end effector can be distinguished with close to 100% accuracy. Our research offers implications for the design of future HRI interfaces, suggesting that motion-based interaction can empower human operators to communicate directly with robots, removing the necessity for additional hardware.

Human-Computer Interaction,Robotics
Gesture Recognition Based on Localist Attractor Networks with Application to Robot Control [Application Notes]

Rui Yan,Keng K. P. Tee,Yuanwei Chua,Haizhou Li,Huajin Tang

DOI: https://doi.org/10.1109/MCI.2011.2176767

IF: 9.809

2012-01-01

IEEE Computational Intelligence Magazine

Abstract:In this work, we proposed an online gesture recognition method based on LAN (Localist Attractor Networks). It was employed to recognize human gestures using streams of feature vectors extracted from real-time sensory data. As an application, the gesture recognition system was used to instruct the robot to execute the predefined commands such as moving in different directions, changing speed, stopping and so on. Experimental results showed a high accuracy in controlling the mobile robot using gesture recognition. This system provides a flexible and easy-to-use human-robot interface to control a robot. The user only needs to demonstrate all gesture patterns a few times before starting the control tasks. The robot is able to memorize all patterns and recognize a given gesture. Furthermore, in order to define a new pattern corresponding to a new control task, users only need to demonstrate this pattern. It is advantageous in the areas of service robots since many of them will be operated by non-expert users.
Vision-Based Real-Time Human–Computer Interaction on Hand Gesture Recognition

Poorvika Singh Negi,Riya Pawar,Roshan Lal

DOI: https://doi.org/10.1007/978-981-15-2329-8_51

2020-01-01

Abstract:Gesture recognition technology has evolved greatly over the years. It is a field in language innovation and development with the aim of deciphering human motions by the use of certain algorithms. In this age of technology, we have fabricated boundless techniques and subsequently seen their drawbacks making us aware of our own limits in terms of speed and naturalness of the human body. The intellect and inventiveness of human beings have led to the development of many tools, gesture recognition technology being one of them. They help us extend the capabilities of our senses by combining natural gestures to operate technology, thereby reducing human efforts and going beyond human abilities. Gestures can be viewed as a route for PCs to start to comprehend human non-verbal communication, in this manner fabricating a more extravagant extension among machines and people than crude content UIs or GUIs, which still breaking point most of contribution to console and mouse and connect normally with no mechanical gadgets. Utilizing the idea of motion acknowledgment, it is conceivable to point a finger now will move as needs be. With the expeditious inventions of three-dimensional applications and the upcoming hype of virtual environments in systems, there is a need of new devices that can sustain these interactions. The advancement of user interface molds and develops the human-computer interaction (HCI). The paper aims to present a review of vision-based hand gesture recognition techniques for HCI and using it to tally finger count using the same.
Interaction and control with the auxiliary of hand gesture

shuai jin,yi li,guangming lu,j luo,weidong chen,xiaoxiang zheng

DOI: https://doi.org/10.1109/ICIST.2011.5765090

2011-01-01

Abstract:This paper presents method used hand gesture recognition in human-computer interaction and control. Nowadays in dataglove-driven motion capture field, researchers preprocess the raw sensor data of the glove with calibration methods for acquiring a high precision in the VR environment. But there are still alternative solutions. Some machine learning algorithms, for example the self-organizing map method, offer a powerful capacity in the case of hand gesture recognition with uncalibrated glove data. We present the reason we introduced the SOM into the recognition process and how it works. Our goal is to construct a stable and robust mechanism to recognize hand gesture and then put it into hand gesture control system as an interaction platform for virtual reality.
Hand gesture recognition and voice controlled robot

M. Meghana,Ch. Usha Kumari,J. Sthuthi Priya,P. Mrinal,K. Abhinav Venkat Sai,S. Prashanth Reddy,K. Vikranth,T. Santosh Kumar,Asisa Kumar Panigrahy

DOI: https://doi.org/10.1016/j.matpr.2020.06.553

2020-01-01

Materials Today: Proceedings

Abstract:This work addresses the robot controlled by hand gesture and voice control. In the technology era, the space between the physical and the digital world is brought closer by the introduction of gesture concept. For all the dangerous tasks we prefer technology rather than people. Even though these robots are being controlled manually in the early stages, these can now be controlled via voice and gestures. This technology of gesture and voice recognition can be defined by the interaction between the computer and the body language of human beings. This constructs the communication link between technology and mankind. The target of this work is to upgrade the complete security to the robot and to simplify the controlling mechanism.
Controlling a Robot Using Leap Motion

Chang Chen,Liang Chen,Xuefeng Zhou,Wu Yan

DOI: https://doi.org/10.1109/icrae.2017.8291351

2017-01-01

Abstract:This paper studies control the robotic arm based on our hand movement and gestures. Data from a Leap Motion controller which detects gestures and hand movement, than the data are delivered through USB to a SCARA robotic arm. Leap Motion, which tracks the movement of hands and a robotic arm is available to control minute movements of the dynamic motions. This paper uses an existing robot, which can be controlled by gestures using a controller device called as Leap Motion. We would like to investigate more on this and implement, such that the robot can be interfaced with a Leap Motion controller for a successful control.
Open Arms: Open-Source Arms, Hands & Control

David Hanson,Alishba Imran,Gerardo Morales,Vytas Krisciunas,Aditya Sagi,Aman Malali,Rushali Mohbe,Raviteja Upadrashta

DOI: https://doi.org/10.48550/arXiv.2205.12992

IF: 3.7

2022-05-20

Robotics

Abstract:Open Arms is a novel open-source platform of realistic human-like robotic hands and arms hardware with 28 Degree-of-Freedom (DoF), designed to extend the capabilities and accessibility of humanoid robotic grasping and manipulation. The Open Arms framework includes an open SDK and development environment, simulation tools, and application development tools to build and operate Open Arms. This paper describes these hands controls, sensing, mechanisms, aesthetic design, and manufacturing and their real-world applications with a teleoperated nursing robot. From 2015 to 2022, the authors have designed and established the manufacturing of Open Arms as a low-cost, high functionality robotic arms hardware and software framework to serve both humanoid robot applications and the urgent demand for low-cost prosthetics, as part of the Hanson Robotics Sophia Robot platform. Using the techniques of consumer product manufacturing, we set out to define modular, low-cost techniques for approximating the dexterity and sensitivity of human hands. To demonstrate the dexterity and control of our hands, we present a Generative Grasping Residual CNN (GGR-CNN) model that can generate robust antipodal grasps from input images of various objects in real-time speeds (22ms). We achieved state-of-the-art accuracy of 92.4% using our model architecture on a standard Cornell Grasping Dataset, which contains a diverse set of household objects.
Hand Gesture Based Robot Control System Using Leap Motion

Sunjie Chen,Hongbin Ma,Chenguang Yang,Mengyin Fu

DOI: https://doi.org/10.1007/978-3-319-22879-2_53

2015-01-01

Abstract:Gesture based human-robot interface is a highly efficient robot control strategy for its simple operation and high availability. This paper develops a hand gesture based robot control system using Leap Motion. The process that the robot responds to human's hand gesture contains noise suppression, coordinate transformation and inverse kinematics. A Client/Server structured robot control system is developed, which provides the function of controlling virtual universal robot UR10 with hand gesture. Finally, experimental results demonstrate that the system is effective and practical.
Predictive Hand Gesture Classification for Real Time Robot Control.

Yuanda Hu,Jiacheng Xu,Zhiqiang Ma,Guitao Cao

DOI: https://doi.org/10.1145/3240876.3240914

2018-01-01

Abstract:In this paper, we propose a vision-based hand gesture recognition system for human-computer interaction. The gesture recognition systems are employed in developing a rock-paper-scissors game between human and our robotic hands in realtime. Our task is to predict the gestures as soon as possible by using high-speed cameras. Due to the computational complexity, the standard long-term recurrent convolution network-based action classification system cannot be contented with classification tasks based on high-speed cameras. We propose to address this issue by employing a more efficient network architecture and using a threshold-based method to predict the gesture in advance. We validate our proposed method on the new gesture dataset for the rock-paper-scissors game. The model is able to successfully learn gestures varying in duration and complexity. A comparative analysis of CNN and long-term recurrent convolution network is performed. We report a gesture classification accuracy of 97% and report a near real-time computational complexity of 7 ms per frame.
Hand Motion-Based Remote Control Interface With Vibrotactile Feedback For Home Robots

Juan Wu,Guifang Qiao,Jun Zhang,Ying Zhang,Guangming Song

DOI: https://doi.org/10.5772/56617

IF: 1.714

2013-01-01

International Journal of Advanced Robotic Systems

Abstract:This paper presents the design and implementation of a hand-held interface system for the locomotion control of home robots. A handheld controller is proposed to implement hand motion recognition and hand motion-based robot control. The handheld controller can provide a 'connect-and-play' service for the users to control the home robot with visual and vibrotactile feedback. Six natural hand gestures are defined for navigating the home robots. A three-axis accelerometer is used to detect the hand motions of the user. The recorded acceleration data are analysed and classified to corresponding control commands according to their characteristic curves. A vibration motor is used to provide vibrotactile feedback to the user when an improper operation is performed. The performances of the proposed hand motion-based interface and the traditional keyboard and mouse interface have been compared in robot navigation experiments. The experimental results of home robot navigation show that the success rate of the handheld controller is 13.33% higher than the PC based controller. The precision of the handheld controller is 15.4% more than that of the PC and the execution time is 24.7% less than the PC based controller. This means that the proposed hand motion-based interface is more efficient and flexible.
An Anthropomorphic Robot Hand Developed Based on Underactuated Mechanism and Controlled by Emg Signals

Da-Peng Yang,Jing-Dong Zhao,Yi-Kun Gu,Xin-Qing Wang,Nan Li,Li Jiang,Hong Liu,Hai Huang,Da-Wei Zhao

DOI: https://doi.org/10.1016/s1672-6529(08)60119-5

IF: 2.995

2009-01-01

Journal of Bionic Engineering

Abstract:When developing a humanoid myo-control hand, not only the mechanical structure should be considered to afford a high dexterity, but also the myoelectric (electromyography, EMG) control capability should be taken into account to fully accomplish the actuation tasks. This paper presents a novel humanoid robotic myocontrol hand (AR hand III) which adopted an underactuated mechanism and a forearm myocontrol EMG method. The AR hand III has five fingers and 15 joints, and actuated by three embedded motors. Underactuation can be found within each finger and between the rest three fingers (the middle finger, the ring finger and the little finger) when the hand is grasping objects. For the EMG control,, two specific methods are proposed: the three-fingered hand gesture configuration of the AR hand III and a pattern classification method of EMG signals based on a statistical learning algorithm - Support Vector Machine (SVM). Eighteen active hand gestures of a testee are recognized effectively, which can be directly mapped into the motions of AR hand III. An on-line EMG control scheme is established based on two different decision functions: one is for the discrimination between the idle and active modes, the other is for the recognition of the active modes. As a result, the AR hand III can swiftly follow the gesture instructions of the testee with a time delay less than 100 ms.
Research of the Gesture Control System for a Industrial Robot

Tiansong Liu

DOI: https://doi.org/10.1109/ICEEMT59522.2023.10263211

2023-07-21

Abstract:At present, the programming methods of industrial robots mainly include off-line programming and instructional programming. However, both of the methods are time-consuming and require experienced robotics technicians. In order to improve the interactivity of industrial robot control system, this paper has designed a control system of industrial robot based on hand gestures. The whole system is composed of four parts. The IMU is used in the human hand gesture acquisition, and the signal processing of IMU is analyzed. The relationship between gestures and robot movements in linear motion mode and joint motion mode are analyzed respectively. The ABB IRB 120 robot is used as a test object and its program is designed. Finally, the effectiveness of the method proposed in this paper is validated.

Computer Science,Engineering
A Real Time Vision-Based Hand Gestures Recognition System

Lei Shi,Yangsheng Wang,Jituo Li

DOI: https://doi.org/10.1007/978-3-642-16493-4_36

2010-01-01

Abstract:Hand gesture recognition is an important aspect in Human-Computer interaction, and can be used in various applications, such as virtual reality and computer games. In this paper, we propose a real time hand gesture recognition system. It includes three major procedures: detection, tracking and recognition. In hand detection stage, an open hand is detected by the histograms of or ented gradient and AdaBoost method. The hand detector is trained by the AdaBoost algorithm with HOG features. A contour based tracker is applied in combining condensation and partitioned sampling. After a hand is detected in the image, the tracker can track the hand contour in real time. During the tracking, the trajectory is saved to perform hand gesture recognition in the last stage. Recognition of the hand moving trajectory is implemented by hidden Markov models. Several HMMs are trained in advance, and the results from the tracking stage are then recognized using the trained HMMs. Experiments have been conducted to validate the performance of the proposed system. Under normal webcam it can recognize the predefined gestures quickly and precisely. As it is easy to develop other hand gestures, the proposed system has good potential in many applications.
Using Deep Learning Technology to Realize the Automatic Control Program of Robot Arm Based on Hand Gesture Recognition

Shang-Liang Chen,Li-Wu Huang

DOI: https://doi.org/10.46604/ijeti.2021.7342

2021-08-09

International Journal of Engineering and Technology Innovation

Abstract:In this study, the robot arm control, computer vision, and deep learning technologies are combined to realize an automatic control program. There are three functional modules in this program, i.e., the hand gesture recognition module, the robot arm control module, and the communication module. The hand gesture recognition module records the user’s hand gesture images to recognize the gestures’ features using the YOLOv4 algorithm. The recognition results are transmitted to the robot arm control module by the communication module. Finally, the received hand gesture commands are analyzed and executed by the robot arm control module. With the proposed program, engineers can interact with the robot arm through hand gestures, teach the robot arm to record the trajectory by simple hand movements, and call different scripts to satisfy robot motion requirements in the actual production environment.
Optimal Control and Grasping for a Robotic Hand with a Non-linked Double Tendon Arrangement

Erick J. Sánchez-Garnica,Liliam Rodríguez-Guerrero,Rocío Ortega-Palacios,Omar Jacobo Santos-Sánchez

DOI: https://doi.org/10.1109/tla.2024.10431547

IF: 0.967

2024-02-13

IEEE Latin America Transactions

Abstract:After comparing different robotic hand projects, a problem is identified: when a finger has a degree of freedom, the hand is unable to grasp irregularly shaped objects. This article proposes a solution. The use of a non-linked doubletendon arrangement in the fingers allows them to have free movement; coupled with the use of Inertial Measurement Units to determine its position, ensures that, despite having one degree of freedom per finger, the hand can effectively grasp irregular objects. Additionally, a web application is developed to control hand movements through voice commands. Finally, due to the necessity for these types of devices to be mobile, an optimal control law is used to minimize energy consumption, thereby increasing autonomy when the hand is powered by batteries. As an additional note, the conducted experiments reveal that the movement of all fingers occurs simultaneously, demonstrating that parallel multitasking programming techniques effectively fulfill that purpose.

engineering, electrical & electronic,computer science, information systems
Tactile-Driven Gentle Grasping for Human-Robot Collaborative Tasks

Christopher J. Ford,Haoran Li,John Lloyd,Manuel G. Catalano,Matteo Bianchi,Efi Psomopoulou,Nathan F. Lepora

DOI: https://doi.org/10.48550/arXiv.2303.09346

IF: 3.7

2023-03-16

Robotics

Abstract:This paper presents a control scheme for force sensitive, gentle grasping with a Pisa/IIT anthropomorphic SoftHand equipped with a miniaturised version of the TacTip optical tactile sensor on all five fingertips. The tactile sensors provide high-resolution information about a grasp and how the fingers interact with held objects. We first describe a series of hardware developments for performing asynchronous sensor data acquisition and processing, resulting in a fast control loop sufficient for real-time grasp control. We then develop a novel grasp controller that uses tactile feedback from all five fingertip sensors simultaneously to gently and stably grasp 43 objects of varying geometry and stiffness, which is then applied to a human-to-robot handover task. These developments open the door to more advanced manipulation with underactuated hands via fast reflexive control using high-resolution tactile sensing.

Real-Time Robotic Hand Control Using Hand Gestures

Real time robotic arm control using hand gestures

Real Time Robotic Arm using Leap Motion Controller

The Manipulation of Real-Time Kinect-Based Robotic Arm Using Double-Hand Gestures

Controlling a remotely located Robot using Hand Gestures in real time: A DSP implementation

Hands-On Robotics: Enabling Communication Through Direct Gesture Control

Gesture Recognition Based on Localist Attractor Networks with Application to Robot Control [Application Notes]

Vision-Based Real-Time Human–Computer Interaction on Hand Gesture Recognition

Interaction and control with the auxiliary of hand gesture

Hand gesture recognition and voice controlled robot

Controlling a Robot Using Leap Motion

Open Arms: Open-Source Arms, Hands & Control

Hand Gesture Based Robot Control System Using Leap Motion

Predictive Hand Gesture Classification for Real Time Robot Control.

Hand Motion-Based Remote Control Interface With Vibrotactile Feedback For Home Robots

An Anthropomorphic Robot Hand Developed Based on Underactuated Mechanism and Controlled by Emg Signals

Research of the Gesture Control System for a Industrial Robot

A Real Time Vision-Based Hand Gestures Recognition System

Using Deep Learning Technology to Realize the Automatic Control Program of Robot Arm Based on Hand Gesture Recognition

Optimal Control and Grasping for a Robotic Hand with a Non-linked Double Tendon Arrangement

Tactile-Driven Gentle Grasping for Human-Robot Collaborative Tasks