Abstract:Sign language is commonly used by deaf or mute people to communicate but requires extensive effort to master. It is usually performed with the fast yet delicate movement of hand gestures, body posture, and even facial expressions. Current Sign Language Recognition (SLR) methods usually extract features via deep neural networks and suffer overfitting due to limited and noisy data. Recently, skeleton-based action recognition has attracted increasing attention due to its subject-invariant and background-invariant nature, whereas skeleton-based SLR is still under exploration due to the lack of hand annotations. Some researchers have tried to use off-line hand pose trackers to obtain hand keypoints and aid in recognizing sign language via recurrent neural networks. Nevertheless, none of them outperforms RGB-based approaches yet. To this end, we propose a novel Skeleton Aware Multi-modal Framework with a Global Ensemble Model (GEM) for isolated SLR (SAM-SLR-v2) to learn and fuse multi-modal feature representations towards a higher recognition rate. Specifically, we propose a Sign Language Graph Convolution Network (SL-GCN) to model the embedded dynamics of skeleton keypoints and a Separable Spatial-Temporal Convolution Network (SSTCN) to exploit skeleton features. The skeleton-based predictions are fused with other RGB and depth based modalities by the proposed late-fusion GEM to provide global information and make a faithful SLR prediction. Experiments on three isolated SLR datasets demonstrate that our proposed SAM-SLR-v2 framework is exceedingly effective and achieves state-of-the-art performance with significant margins. Our code will be available at https://github.com/jackyjsy/SAM-SLR-v2

Global-local Enhancement Network for NMFs-aware Sign Language Recognition

Global-Local Enhancement Network for NMF-Aware Sign Language Recognition

StepNet: Spatial-temporal Part-aware Network for Isolated Sign Language Recognition

Natural Language-Assisted Sign Language Recognition

Attention-Based 3D-Cnns for Large-Vocabulary Sign Language Recognition.

TMS-Net: A multi-feature multi-stream multi-level information sharing network for skeleton-based sign language recognition

SF-Net: Structured Feature Network for Continuous Sign Language Recognition

(2+1)D-SLR: an Efficient Network for Video Sign Language Recognition.

Multi-scale Context-Aware Network for Continuous Sign Language Recognition

Multimodal Spatiotemporal Networks for Sign Language Recognition

Multi-View Spatial-Temporal Network for Continuous Sign Language Recognition

Difference-guided multi-scale spatial-temporal representation for sign language recognition

Sign Language Recognition via Skeleton-Aware Multi-Model Ensemble

Two-Stream Network for Sign Language Recognition and Translation

RNN-Transducer Based Chinese Sign Language Recognition

Hand-Model-Aware Sign Language Recognition

Sign Language Recognition Based on Adaptive Hmms with Data Augmentation

Sign Language Translation with Hierarchical Spatio-TemporalGraph Neural Network

Multi-Stream Keypoint Attention Network for Sign Language Recognition and Translation

Sign language recognition using real-sense