Abstract:Automatic sign language recognition (SLR) stands as a vital aspect within the realms of human–computer interaction and computer vision, facilitating the conversion of hand signs utilized by individuals with significant hearing and speech impairments into equivalent text or voice. Researchers have recently used hand skeleton joint information instead of the image pixel due to light illumination and complex background-bound problems. However, besides the hand information, body motion and facial gestures play an essential role in expressing sign language emotion. Also, a few researchers have been working to develop an SLR system by taking a multi-gesture dataset, but their performance accuracy and time complexity are not sufficient. In light of these limitations, we introduce a spatial and temporal attention model amalgamated with a general neural network designed for the SLR system. The main idea of our architecture is first to construct a fully connected graph to project the skeleton information. We employ self-attention mechanisms to extract insights from node and edge features across spatial and temporal domains. Our architecture bifurcates into three branches: a graph-based spatial branch, a graph-based temporal branch, and a general neural network branch, which collectively synergize to contribute to the final feature integration. Specifically, the spatial branch discerns spatial dependencies, while the temporal branch amplifies temporal dependencies embedded within the sequential hand skeleton data. Further, the general neural network branch enhances the architecture's generalization capabilities, bolstering its robustness. In our evaluation, utilizing the Mexican Sign Language (MSL), Pakistani Sign Language (PSL) datasets, and American Sign Language Large Video dataset which comprises 3D joint coordinates for face, body, and hands that conducted experiments on individual gestures and their combinations. Impressively, our model demonstrated notable efficacy, achieving an accuracy rate of 99.96% for the MSL dataset, 92.00% for PSL, and 26.00% for the ASLLVD dataset, which includes more than 2700 classes. These exemplary performance metrics, coupled with the model's computationally efficient profile, underscore its preeminence compared to contemporaneous methodologies in the field.

Real-time sign language recognition based on YOLO algorithm

Real Time American Sign Language Detection Using Yolo-v9

ShuffleNetv2-YOLOv3: a real-time recognition method of static sign language based on a lightweight network

Sign Language Recognition based on YOLOv5 Algorithm for the Telugu Sign Language

Intelligent real-life key-pixel image detection system for early Arabic sign language learners

Real-Time Detection and Identification of Suspects in Forensic Imagery Using Advanced YOLOv8 Object Recognition Models

HGR-FYOLO: a robust hand gesture recognition system for the normal and physically impaired person using frozen YOLOv5

ADDSL: Hand Gesture Detection and Sign Language Recognition on Annotated Danish Sign Language

Deep Learning Based Traffic Sign Recognition Using YOLO Algorithm

Enhancing Bidirectional Sign Language Communication: Integrating YOLOv8 and NLP for Real-Time Gesture Recognition & Translation

Real-Time Sign Language Fingerspelling Recognition using Convolutional Neural Network

Gesture Detection and Recognition Based on Object Detection in Complex Background

Spatial–temporal attention with graph and general neural network-based sign language recognition

TSR-YOLO: A Chinese Traffic Sign Recognition Algorithm for Intelligent Vehicles in Complex Scenes

Sign language recognition using modified deep learning network and hybrid optimization: a hybrid optimizer (HO) based optimized CNNSa-LSTM approach

AUTSL: A Large Scale Multi-modal Turkish Sign Language Dataset and Baseline Methods

Computer Interactive Gesture Recognition Model Based on Improved YOLOv5 Algorithm

Traffic Sign Detection Based on the Improved YOLOv5

Enhancing Signer-Independent Recognition of Isolated Sign Language through Advanced Deep Learning Techniques and Feature Fusion

Albanian Sign Language (AlbSL) Number Recognition from Both Hand's Gestures Acquired by Kinect Sensors