Abstract:Many sign languages are bona fide natural languages with grammatical rules and lexicons hence can benefit from machine translation methods. Similarly, since sign language is a visual-spatial language, it can also benefit from computer vision methods for encoding it. With the advent of deep learning methods in recent years, significant advances have been made in natural language processing (specifically neural machine translation) and in computer vision methods (specifically image and video captioning). Researchers have therefore begun expanding these learning methods to sign language understanding. Sign language interpretation is especially challenging, because it involves a continuous visual-spatial modality where meaning is often derived based on context. The focus of this article, therefore, is to examine various deep learning–based methods for encoding sign language as inputs, and to analyze the efficacy of several machine translation methods, over three different sign language datasets. The goal is to determine which combinations are sufficiently robust for sign language translation without any gloss-based information. To understand the role of the different input features, we perform ablation studies over the model architectures (input features + neural translation models) for improved continuous sign language translation. These input features include body and finger joints, facial points, as well as vector representations/embeddings from convolutional neural networks. The machine translation models explored include several baseline sequence-to-sequence approaches, more complex and challenging networks using attention, reinforcement learning, and the transformer model. We implement the translation methods over multiple sign languages—German (GSL), American (ASL), and Chinese sign languages (CSL). From our analysis, the transformer model combined with input embeddings from ResNet50 or pose-based landmark features outperformed all the other sequence-to-sequence models by achieving higher BLEU2-BLEU4 scores when applied to the controlled and constrained GSL benchmark dataset. These combinations also showed significant promise on the other less controlled ASL and CSL datasets.

Benchmarking deep neural network approaches for Indian Sign Language recognition

Image-based Indian Sign Language Recognition: A Practical Review using Deep Neural Networks

Indian Sign Language Gesture Recognition using Image Processing and Deep Learning

Recognition of Indian Sign Language (ISL) Using Deep Learning Model

Deepsign: Sign Language Detection and Recognition Using Deep Learning

Real-time sign language detection: Empowering the disabled community

Deep Neural Network-Based Sign Language Recognition: A Comprehensive Approach Using Transfer Learning with Explainability

Automated Sign to Speech Conversion Model using Deep Learning

Word recognition from Indian Sign Language in videos using dual feature descriptor and GMT-MASKRCNN recognition technique

iSign: A Benchmark for Indian Sign Language Processing

Sign Language Recognition System using TensorFlow Object Detection API

A Comparative Analysis of Techniques and Algorithms for Recognising Sign Language

Android-Based Application for Real-Time Indonesian Sign Language Recognition Using Convolutional Neural Network

Real-time Sign Language Recognition Using MobileNetV2 and Transfer Learning

Sign Language Recognition from Digital Videos Using Deep Learning Methods

Hierarchical Windowed Graph Attention Network and a Large Scale Dataset for Isolated Indian Sign Language Recognition

Indian Sign Language Recognition Using Mediapipe Holistic

Dynamic gesture recognition using hand pose-based neural networks for sign language interpretation

Indian Sign Language recognition system using SURF with SVM and CNN

Hypertuned Deep Convolutional Neural Network for Sign Language Recognition

Deep Learning Methods for Sign Language Translation