Sign Annotation Generation to Alphabets Via Integrating Visual Data with Somatosensory Data from Flexible Strain Sensor-Based Data Glove

Yangyang Zhang,Weijun Xu,Xia Zhang,Liping Li

DOI: https://doi.org/10.1016/j.measurement.2022.111700

IF: 5.6

2022-01-01

Measurement

Abstract:Sign language forms a significant way of communication for deaf-mute people. There are different types of sign languages of different countries in different regions. The sign language recognition system translates the gestures into the alphabet, which solves a barrier in comprehension of conversation. Visual techniques and somatosensory strain sensors are both useful modalities to detect sign gesture. In order to improve the recognition accuracy and wear resistance, this paper proposes a real-time alphabets conversion system, which combines the visual data with the data glove based on flexible strain sensor and the somatosensory data of the camera, supplemented by deep learning. In this learning structure, convolutional neural network is used to process the visual data, and then the sparse neural network for sensor data fusion and recognition is realized in the feature layer. A module is proposed to train the linking of hand motions primitives to alphabets (A-Z). The proposed framework is tested on a dataset of sign and descriptive alphabets. The proposed system would recognize the 26 letters with a recognition rate of up to 99.50% and a recognition time of less than 1 s, which helps to understand observations of sign as words and texts.