Abstract:Many sign languages are bona fide natural languages with grammatical rules and lexicons hence can benefit from machine translation methods. Similarly, since sign language is a visual-spatial language, it can also benefit from computer vision methods for encoding it. With the advent of deep learning methods in recent years, significant advances have been made in natural language processing (specifically neural machine translation) and in computer vision methods (specifically image and video captioning). Researchers have therefore begun expanding these learning methods to sign language understanding. Sign language interpretation is especially challenging, because it involves a continuous visual-spatial modality where meaning is often derived based on context. The focus of this article, therefore, is to examine various deep learning–based methods for encoding sign language as inputs, and to analyze the efficacy of several machine translation methods, over three different sign language datasets. The goal is to determine which combinations are sufficiently robust for sign language translation without any gloss-based information. To understand the role of the different input features, we perform ablation studies over the model architectures (input features + neural translation models) for improved continuous sign language translation. These input features include body and finger joints, facial points, as well as vector representations/embeddings from convolutional neural networks. The machine translation models explored include several baseline sequence-to-sequence approaches, more complex and challenging networks using attention, reinforcement learning, and the transformer model. We implement the translation methods over multiple sign languages—German (GSL), American (ASL), and Chinese sign languages (CSL). From our analysis, the transformer model combined with input embeddings from ResNet50 or pose-based landmark features outperformed all the other sequence-to-sequence models by achieving higher BLEU2-BLEU4 scores when applied to the controlled and constrained GSL benchmark dataset. These combinations also showed significant promise on the other less controlled ASL and CSL datasets.

Deep Learning Based Natural Language Processing for End to End Speech Translation

Recent Trends in Deep Learning Based Natural Language Processing

Deep Learning Based TTS-STT Model with Transliteration for Indic Languages

A Review on Deep Learning Based Lip-Reading

Deep Learning for Hindi Text Classification: A Comparison

A Review of Deep Learning Techniques for Speech Processing

Deep Learning for Indian Language Processing: A Focus on Speech Recognition Systems

Deep Learning in Natural Language Processing: A State-of-the-Art Survey

Development of an End-to-End Deep Learning Framework for Sign Language Recognition, Translation, and Video Generation

Deep Learning and Machine Learning -- Natural Language Processing: From Theory to Application

Recent Trends in Deep Learning Based Natural Language Processing [Review Article]

Deep Learning for Natural Language Processing: A Survey

Deep Learning and Its Applications to Natural Language Processing

Enhancing Lip Reading: A Deep Learning Approach with CNN and RNN Integration

Advancements in Named Entity Recognition using Deep Learning Techniques: A Comprehensive Study on Emerging Trends

An introduction to Deep Learning in Natural Language Processing: Models, techniques, and tools

A Comparative Study on End-to-end Speech to Text Translation

Comprehensive analysis of natural language processing

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Deep Learning Methods for Sign Language Translation

Natural Language Processing Advancements By Deep Learning: A Survey