Hand-Gesture Based Text Input for Wearable Computers

Yang Liu,Xiabi Liu,Yunde Jia

DOI: https://doi.org/10.1109/ICVS.2006.34

2006-01-01

Abstract:In this paper, we present a case study that exemplifies general ideas of system integration and coordination. The application field of assistant technology provides an ideal test bed for complex computer vision systems including real-time components, ...

What problem does this paper attempt to address?

COMTIS: Customizable Touchless Interaction System for Large Screen Visualization

Jiaxin Liu,Hongxin Zhang,Chuankang Li

DOI: https://doi.org/10.1016/j.vrih.2020.01.003

2020-01-01

Virtual Reality & Intelligent Hardware

Abstract:Background: Large screen visualization systems have been widely utilized in many industries. Such systems can help illustrate the working states of different production systems. However, efficient interaction with such systems is still a focus of related research. Methods: In this paper, we propose a touchless interaction system based on RGB-D camera using a novel bone-length constraining method. The proposed method optimizes the joint data collected from RGB-D cameras with more accurate and more stable results on very noisy data. The user can customize the system by modifying the finite-state machine in the system and reuse the gestures in multiple scenarios, reducing the number of gestures that need to be designed and memorized. Results/Conclusions: he authors tested the system in two cases. In the first case, we illustrated a process in which we improved the gesture designs on our system and tested the system through user study. In the second case, we utilized the system in the mining industry and conducted a user study, where users say that they think the system is easy to use.
Hand Tracking and Gesture Recognition Method Based on Wearable Vision

LIU Yang,JIA Yun-de

DOI: https://doi.org/10.3969/j.issn.1001-0645.2005.12.012

2005-01-01

Abstract:In order to solve natural interactive problems between human and wearable computers,a hand tracking and gesture recognition method based on wearable vision is proposed.The method is an extension of the Icondensation algorithm,which integrates the shape and depth information provided by a wearable vision system for robust hand tracking.Furthermore,gesture transition model is introduced to recognize gestures.The method can automatically switch between multiple pre-defined gestures in the tracking loop.Experimental results showed that the proposed method worked well in dynamic and complex background.The method can provide natural gesture-based interfaces for wearable computing systems.
A robust hand tracking and gesture recognition method for wearable visual interfaces and its applications.

Yang Liu,Yunde Jia

DOI: https://doi.org/10.1109/ICIG.2004.24

2004-01-01

Abstract:Gesture-based interface is one of the most promising modes of human-computer interaction for wearable computers. This paper proposes a robust hand tracking and gesture recognition method for wearable visual interfaces, which is an extension of ICONDENSATION algorithm. The method integrates shape and depth information for robust hand tracking. Gesture recognition is realized through the maximum posterior estimation of several pre-defined gestures. The experimental results show that the proposed method works well in dynamic and complex background. Several promising applications in wearable computers are also discussed.
WristCam: A Wearable Sensor for Hand Trajectory Gesture Recognition and Intelligent Human–Robot Interaction

Feiyu Chen,Honghao Lv,Zhibo Pang,Junhui Zhang,Yonghong Hou,Ying Gu,Huayong Yang,Geng Yang

DOI: https://doi.org/10.1109/jsen.2018.2877978

IF: 4.3

2019-01-01

IEEE Sensors Journal

Abstract:As a promising component for body sensor networks, the wearable sensors for hand gesture recognition have increasingly received great attention in recent years. By interpreting human intentions through hand gestures, the natural human-robot interaction can be realized in the smart home where the youth and the elderly can perform hand gestures to control the household robot or the robotic wheelchair. Here, a wearable wrist-worn camera sensor was shown to recognize hand trajectory gestures. The moving velocity of the user's hand was deduced from the matched speeded up robust features keypoints of the moving background of the video sequence. Furthermore, the segmentation of continuous gestures was achieved by detecting the predefined gesture starting signal from the hand region of the image, which was segmented by the lazy snapping algorithm. In this paper, 10 types of gestures and 1350 gesture samples collected from 15 subjects at three different scenes were classified by the dynamic time warping algorithm and the results achieved an average recognition accuracy up to 97.6%. Moreover, the practicability of the proposed system was further demonstrated by controlling a cooperative robot to draw letters on paper.
Interaction and control with the auxiliary of hand gesture

shuai jin,yi li,guangming lu,j luo,weidong chen,xiaoxiang zheng

DOI: https://doi.org/10.1109/ICIST.2011.5765090

2011-01-01

Abstract:This paper presents method used hand gesture recognition in human-computer interaction and control. Nowadays in dataglove-driven motion capture field, researchers preprocess the raw sensor data of the glove with calibration methods for acquiring a high precision in the VR environment. But there are still alternative solutions. Some machine learning algorithms, for example the self-organizing map method, offer a powerful capacity in the case of hand gesture recognition with uncalibrated glove data. We present the reason we introduced the SOM into the recognition process and how it works. Our goal is to construct a stable and robust mechanism to recognize hand gesture and then put it into hand gesture control system as an interaction platform for virtual reality.
A robust hand tracking for gesture-based interaction of wearable computers

Yang Liu,Yunde Jia

DOI: https://doi.org/10.1109/ISWC.2004.4

2004-01-01

Abstract:Hand gesture-based interface is one of the most promising modes of natural and fluid human-computer interaction that substitutes for the mouse and keyboard in wearable computing systems. In wearable computing scenarios, hand positioning and tracking is particularly difficult due to complex background, lighting variation and image dithering caused by head movement. This paper proposes a robust hand tracking method for gesture-based interaction of a wearable computer with a visual helmet. The method is an extension of the basic CONDENSATION algorithm, which is able to track hand in dynamic and complex background. Furthermore, the algorithm can recognize the current hand gesture and automatically switch between multiple well-defined gesture templates in the tracking loop. The experimental results show that the proposed algorithm worked well in dynamic and complex background in real time.
An interaction system using mixed hand gestures.

Zhong Yang,Yi Li,Yang Zheng,Weidong Chen,Xiaoxiang Zheng

DOI: https://doi.org/10.1145/2350046.2350074

2012-01-01

Abstract:This paper presents a mixed hand gesture interaction system in virtual environment, in which "mixed" means static and dynamic hand gestures are combined for both navigation and object manipulation. Firstly, a simple average background model and skin color are used for hand area segmentation. Then a state-based spotting algorithm is employed to automatically identify two types of hand gestures. A voting-based method is used for quick classification of static gestures. And we use the hidden Markov model (HMM) to recognize dynamic gestures. Since the training of HMM requires the consistency of the training data, outputted by the feature extraction, a data aligning algorithm is raised. Through our mixed hand gesture system, users can perform complicated operating commands in a natural way. The experimental results demonstrate that our methods are effective and accurate.
Ihand: Hand Recognition-Based Text Input Method for Wearable Devices

Qiang Chu,Chao Ping Chen,Haiyang Hu,Xiaojun Wu,Baoen Han

DOI: https://doi.org/10.3390/computers13030080

2024-01-01

Computers

Abstract:Text input using hand gestures is an essential component of human–computer interaction technology, providing users with a more natural and enriching interaction experience. Nevertheless, the current gesture input methods have a variety of issues, including a high learning cost for users, poor input performance, and reliance on hardware. To solve these problems and better meet the interaction requirements, a hand recognition-based text input method called iHand is proposed in this paper. In iHand, a two-branch hand recognition algorithm combining a landmark model and a lightweight convolutional neural network is used. The landmark model is used as the backbone network to extract hand landmarks, and then an optimized classification head, which can preserve the space structure of landmarks, is designed to classify gestures. When the landmark model fails to extract hand landmarks, a lightweight convolutional neural network is employed for classification. Regarding the way letters are entered, to reduce the learning cost, the sequence of letters is mapped as a two-dimensional layout, and users can type with seven simple hand gestures. Experimental results on the public datasets show that the proposed hand recognition algorithm achieves high robustness compared to state-of-the-art approaches. Furthermore, we tested the performance of users’ initial use of iHand for text input. The results showed that the iHand’s average input speed was 5.6 words per minute, with the average input error rate was only 1.79%.
Design and evaluation of a hand gesture recognition approach for real-time interactions

Vaidyanath Areyur Shanthakumar,Chao Peng,Jeffrey Hansberger,Lizhou Cao,Sarah Meacham,Victoria Blakely

DOI: https://doi.org/10.1007/s11042-019-08520-1

IF: 2.577

2020-02-21

Multimedia Tools and Applications

Abstract:Hand gestures are a natural and intuitive form for human-environment interaction and can be used as an input alternative in human-computer interaction (HCI) to enhance usability and naturalness. Many existing approaches have employed vision -based systems to detect and recognize hand gestures. However, vision-based systems usually require users to move their hands within restricted space, where the optical device can capture the motion of hands. Also, vision-based systems may suffer from self-occlusion issues due to sophisticated finger movements. In this work, we use a sensor-based motion tracking system to capture 3D hand and finger motions. To detect and recognize hand gestures, we propose a novel angular-velocity method, which is directly applied to real-time 3D motion data streamed by the sensor-based system. Our approach is capable of recognizing both static and dynamic gestures in real-time. We assess the recognition accuracy and execution performance with two interactive applications that require gesture input to interact with the virtual environment. Our experimental results show high recognition accuracy, high execution performance, and high-levels of usability.

computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering
A Hand Gesture Recognition Framework and Wearable Gesture-Based Interaction Prototype for Mobile Devices.

Zhiyuan Lu,Xiang Chen,Qiang Li,Xu Zhang,Ping Zhou

DOI: https://doi.org/10.1109/THMS.2014.2302794

2014-01-01

IEEE Transactions on Human-Machine Systems

Abstract:An algorithmic framework is proposed to process acceleration and surface electromyographic (SEMG) signals for gesture recognition. It includes a novel segmentation scheme, a score-based sensor fusion scheme, and two new features. A Bayes linear classifier and an improved dynamic time-warping algorithm are utilized in the framework. In addition, a prototype system, including a wearable gesture sens...
Design of hand gesture recognition system for human-computer interaction

Tsung-Han Tsai,Chih-Chi Huang,Kung-Long Zhang

DOI: https://doi.org/10.1007/s11042-019-08274-w

IF: 2.577

2019-12-12

Multimedia Tools and Applications

Abstract:Human-Computer interaction (HCI) with gesture recognition is designed to recognize a number of meaningful human expressions, and has become a valuable and intuitive computer input technique. Hand gestures are one of the most intuitive and common forms of communication, and can communicate a wide range of meaning. Vision-based hand gesture recognition has received a significant amount of research attention in recent years. However, the field still presents a number of challenges for researchers. In the vision-based hand gesture interaction process between humans and computers, gesture interpretation must be performed quickly and with high accuracy. In this paper, a low-cost HCI system with hand gesture recognition is proposed. This system uses several vision techniques. Skin and motion detection is used for capturing the region-of-interest from the background regions. A connected component labeling algorithm is proposed to identify the centroid of an object. To identify the exact area of hand gesture, the arm area is removed with the aid of a convex hull algorithm. Moreover, a real-time demonstration system is developed, based on a single-camera mechanism which allows for the use of wearable devices. Simulation results show that the recognition rate is still high, although some interference is encountered in the simulated environments.

computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering
Designing and Evaluating Hand-to-Hand Gestures with Dual Commodity Wrist-Worn Devices

Yiqin Lu,Bingjian Huang,Chun Yu,Guahong Liu,Yuanchun Shi

DOI: https://doi.org/10.1145/3380984

2020-01-01

Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies

Abstract:Hand gestures provide a natural and easy-to-use way to input commands. However, few works have studied the design space of bimanual hand gestures or attempted to infer gestures that involve devices on both hands. We explore the design space of hand-to-hand gestures, a group of gestures that are performed by touching one hand with the other hand. Hand-to-hand gestures are easy to perform and provide haptic feedback on both hands. Moreover, hand-to-hand gestures generate simultaneous vibration on two hands that can be sensed by dual off-the-shelf wrist-worn devices. In this work, we derive a hand-to-hand gesture vocabulary with subjective ratings from users and select gesture sets for real-life scenarios. We also take advantage of devices on both wrists to demonstrate their gesture-sensing capability. Our results show that the recognition accuracy for fourteen gestures is 94.6% when the user is stationary, and the accuracy for five gestures is 98.4% or 96.3% when the user is walking or running, respectively. This is significantly more accurate than a single device worn on either wrist. Our further evaluation also validates that users can easily remember hand-to-hand gestures and use our technique to invoke commands in real-life contexts.
A prototype of gesture-based interface

Zhiyuan Lu,Xiang Chen,Zhangyan Zhao,Kongqiao Wang

DOI: https://doi.org/10.1145/2037373.2037380

2011-01-01

Abstract:This paper introduces a novel gesture-based human-machine interface prototype, which consists of a wearable belt embedding with four surface electromyography (SEMG) sensors, a tri-axis accelerometer and an application program running on NOKIA 5800XM. The sensor belt captures hand gestures by acquiring SEMG and acceleration (ACC) signals from forearm, and sends the m out via Bluetooth. The application program receives the data and translates them into control commands of a given interaction application. Experimental results of two test schemes conducted on hand gesture recognition and media player operation demonstrate the validity of the proposed gesture-based interface prototype.
Wearable Human-machine Gesture Interaction Based on Fabric Piezoelectric Sensor

Yiming Lu,Zewen Li,Xinwang Wang,Jiashun Jiang,Mingzhu Zhu,Mengying Xie

DOI: https://doi.org/10.1109/jsen.2024.3416188

IF: 4.3

2024-01-01

IEEE Sensors Journal

Abstract:Gesture recognition has great application prospects in the field of human-machine interaction (HMI). To date, the development of wearable and cost-effective sensors for gesture recognition remains a challenge. In this study, we propose a comprehensive system solution for gesture recognition and HMI-based on electrospun polyvinylidene fluoride (PVDF)/barium titanate (BaTiO3) nanocomposite membrane sensors. The sensor comprising three types of fabric can effectively recognize different bending angles and exhibits reliable repeatability. The gesture classification recognition based on the sensor exhibits high accuracy in three machine learning algorithms [K nearest neighbor (KNN), support vector machine (SVM), and neural network (NN)], with accuracy rates of 96.7%, 97.9%, and 97.8% achieved over 5000 groups of ten different gestures. Additionally, we design a scheme for real-time remote control of a mechanical claw based on this sensor. The results demonstrate that the proposed solution provides a promising platform for gesture recognition and HMI.
Self‐Powered Gesture Recognition Wristband Enabled by Machine Learning for Full Keyboard and Multicommand Input

Puchuan Tan,Xi Han,Yang Zou,Xuecheng Qu,Jiangtao Xue,Tong Li,Yiqian Wang,Ruizeng Luo,Xi Cui,Yuan Xi,Le Wu,Bo Xue,Dan Luo,Yubo Fan,Xun Chen,Zhou Li,Zhong Lin Wang

DOI: https://doi.org/10.1002/adma.202200793

IF: 29.4

2022-04-27

Advanced Materials

Abstract:Virtual reality is a brand-new technology that can be applied extensively. To realize virtual reality, certain types of human-computer interaction equipment are necessary. Existing virtual reality technologies often rely on cameras, data gloves, game pads, and other equipment. These equipment are either bulky, inconvenient to carry and use, or expensive to popularize. Therefore, the development of a convenient and low-cost high-precision human-computer interaction device can contribute positively to the development of virtual reality technology. In this study, a gesture recognition wristband that can realize a full keyboard and multicommand input is developed. The wristband is convenient to wear, low in cost, and does not affect other daily operations of the hand. This wristband is based on physiological anatomy as well as aided by active sensor and machine learning technology; it can achieve a maximum accuracy of 92.6% in recognizing 26 letters. This wristband offers broad application prospects in the fields of gesture command recognition, assistive devices for the disabled, and wearable electronics.

materials science, multidisciplinary,chemistry, physical,physics, applied, condensed matter,nanoscience & nanotechnology
Adaptive and Personalized Gesture Recognition Using Textile Capacitive Sensor Arrays

Alexander Nelson,Gurashish Singh,Ryan Robucci,Chintan Patel,Nilanjan Banerjee

DOI: https://doi.org/10.1109/tmscs.2015.2495100

2015-04-01

Abstract:Upper extremity mobility impairment is a common sequel of Spinal Cord Injury (SCI), brain injury, strokes, and degenerative diseases such as Guillain-Barre and ALS. Existing assistive technology solutions that provide access as user input devices are intrusive and expensive, and require physical contact that can have deleterious effects such as skin friction injury for paralyzed users who have reduced skin sensitivity. To address this problem, in this paper, we present the design, implementation, and evaluation of a non-contact proximity gesture recognition system using fabric capacitive sensor arrays. The fabric sensors are lightweight, flexible, and can be easily integrated into items of quotidian use such as clothing, bed sheets, and pillow covers. Our gesture recognition algorithm builds on two known classification techniques, Hidden Markov Model and Dynamic Time Warping to convert raw capacitance values to alphanumeric gestures. Our system is personalized to the user, allowing personalized selection of gesture sets and definition of gesture patterns in accordance with their capabilities. Our system adapts to changes in sensor configuration and orientation with minimal user training and intervention. We have evaluated our system in the context of a gesture-driven home automation system on six subjects that includes an individual who has a C6 Spinal Cord injury. We show that our system can recognize gestures of varying complexity with an average accuracy of 99 percent with minimal training.

English Else
Contactless Human–computer Interaction System Based on Three-Dimensional Holographic Display and Gesture Recognition

Yixin Yang,Yunhui Gao,Kexuan Liu,Zehao He,Liangcai Cao

DOI: https://doi.org/10.1007/s00340-023-08128-2

2023-01-01

Applied Physics B

Abstract:Contactless interaction provides a hygienic, safe, and intelligent way for communication between human and computers. As computing systems continue to advance, there is a growing focus on input methods and keyboard interaction approaches. However, current research on virtual keyboard systems remains confined to two-dimensional interfaces. The existing contactless interaction implementations are generally limited by handheld tools, wearable devices, or projection screens. In this work, we propose a contactless three-dimensional (3D) virtual keyboard system, which combines gesture recognition and holographic display. A hand-tracking sensor is used to collect the user’s gestures and fingertip positions as input. A spatial light modulator is utilized to generate the corresponding holographic 3D display images. The hand-tracking sensor and display module are controlled in a synchronous manner by a personal computer. The robust and real-time system is demonstrated through user-interactive experiments with a 3D virtual keyboard. The scalability of the proposed human–computer interaction system supports a wide range of applications, including education, communication, disease prevention, information security, and virtual reality.
A Multimodal Labeling Interface For Wearable Computing

Shanqing Li,Yunde Jia

DOI: https://doi.org/10.1145/1719970.1720027

2010-01-01

Abstract:Under wearable environments, it is not convenient to label an object with portable keyboards and mice. This paper presents a multimodal labeling interface to solve this problem with natural and efficient operations. Visual and audio modalities cooperate with each other: an object is encircled by visual tracking of a pointing gesture, and meanwhile its name is obtained by speech recognition. In this paper, we propose a concept of virtual touchpad based on stereo vision techniques. With the touchpad, the object encircling task is achieved by drawing a closed curve on a transparent blackboard. The touch events and movements of a pointing gesture are robustly detected for natural gesture interactions. The experimental results demonstrate the efficiency and usability of our multimodal interface.
Recognizing Hand Gestures with Pressure-Sensor-Based Motion Sensing.

Yufei Zhang,Bin Liu,Zhiqiang Liu

DOI: https://doi.org/10.1109/tbcas.2019.2940030

2019-01-01

IEEE Transactions on Biomedical Circuits and Systems

Abstract:This paper proposes a novel framework to process pressure signals for real-time and robust gesture recognition, which includes an innovative segmentation scheme, a gesture recognition scheme and a pressure-parameter adaptive updating strategy. A prototype system, including a wearable gesture sensing device with four pressure sensors and the corresponding algorithmic framework, is developed to realize real-time gesture-based interaction. With the device worn on the wrist, the user can interact with the computer using 8 predefined gestures. Experimental results show that the delay of gesture recognition is about 100 ms, with the average accuracy of 95.28 in the experienced-user test and 86.20 in the inexperienced-user test. Finally, the system is evaluated by a mouse-controlling interaction task and performs well. Both experienced and inexperienced people can easily and quickly complete interactive tasks. These results demonstrate that a pressure-sensor based wristband can be used to classify hand gestures well and to control the mouse interaction. This approach provides an interactive way to replace the mouse for decreasing the risk of the carpal tunnel syndrome (CTS).
Hand Gesture Recognition and Real-time Game Control Based on A Wearable Band with 6-Axis Sensors.

Yande Li,TaiMan Wang,Aamir Khan,Lian Li,Caihong Li,Yi Yang,Li Liu

DOI: https://doi.org/10.1109/ijcnn.2018.8489743

2018-01-01

Abstract:Human-computer interaction introduces critical open door with the proceeds with improvement of wearable gadgets. Gesture recognition through smart devices is becoming a popular research direction. This paper proposes a hand gesture recognition and real-time game control system that is capable of continues human-computer interaction in view of an off-the-rack business wearable wristband. We utilize three-axis accelerator and gyroscope sensors embedded in smart band to collect hand motion information and use Kinect camera capture video information for manual segmentation during the training model phase. A continuous gesture segmentation algorithm based on sliding window and DTW algorithm is developed to detect meaningful gestures in the real-time game control stage. In addition, an android game named Fly Birds which is controlled by gesture recognition result is presented to simulate real-time human-computer interaction. Then, we classify the data in the window using common classifiers. Finally, our experimental results show that, we can accurately identify the designed gestures during the stage of static gesture recognition, and we also achieve a perfect interactive effect in the process of dynamic real-time game control. The experiment outcomes will advance the ascent of human-PC cooperation in view of hand gesture recognition and related applications will rise in vast numbers.

Hand-Gesture Based Text Input for Wearable Computers

COMTIS: Customizable Touchless Interaction System for Large Screen Visualization

Hand Tracking and Gesture Recognition Method Based on Wearable Vision

A robust hand tracking and gesture recognition method for wearable visual interfaces and its applications.

WristCam: A Wearable Sensor for Hand Trajectory Gesture Recognition and Intelligent Human–Robot Interaction

Interaction and control with the auxiliary of hand gesture

A robust hand tracking for gesture-based interaction of wearable computers

An interaction system using mixed hand gestures.

Ihand: Hand Recognition-Based Text Input Method for Wearable Devices

Design and evaluation of a hand gesture recognition approach for real-time interactions

A Hand Gesture Recognition Framework and Wearable Gesture-Based Interaction Prototype for Mobile Devices.

Design of hand gesture recognition system for human-computer interaction

Designing and Evaluating Hand-to-Hand Gestures with Dual Commodity Wrist-Worn Devices

A prototype of gesture-based interface

Wearable Human-machine Gesture Interaction Based on Fabric Piezoelectric Sensor

Self‐Powered Gesture Recognition Wristband Enabled by Machine Learning for Full Keyboard and Multicommand Input

Adaptive and Personalized Gesture Recognition Using Textile Capacitive Sensor Arrays

Contactless Human–computer Interaction System Based on Three-Dimensional Holographic Display and Gesture Recognition

A Multimodal Labeling Interface For Wearable Computing

Recognizing Hand Gestures with Pressure-Sensor-Based Motion Sensing.

Hand Gesture Recognition and Real-time Game Control Based on A Wearable Band with 6-Axis Sensors.