Abstract:Deep-learning-based continuous sign language recognition (CSLR) models typically consist of a visual module, a sequential module, and an alignment module. However, the effectiveness of training such CSLR backbones is hindered by limited training samples, rendering the use of a single connectionist temporal classification loss insufficient. To address this limitation, we propose three auxiliary tasks to enhance CSLR backbones. First, we enhance the visual module, which is particularly sensitive to the challenges posed by limited training samples, from the perspective of consistency. Specifically, since sign languages primarily rely on signers’ facial expressions and hand movements to convey information, we develop a keypoint-guided spatial attention module that directs the visual module to focus on informative regions, thereby ensuring spatial attention consistency. Furthermore, recognizing that the output features of both the visual and sequential modules represent the same sentence, we leverage this prior knowledge to better exploit the power of the backbone. We impose a sentence embedding consistency constraint between the visual and sequential modules, enhancing the representation power of both features. The resulting CSLR model, referred to as consistency-enhanced CSLR, demonstrates superior performance on signer-dependent datasets, where all signers appear during both training and testing. To enhance its robustness for the signer-independent setting, we propose a signer removal module based on feature disentanglement, effectively eliminating signer-specific information from the backbone. To validate the effectiveness of the proposed auxiliary tasks, we conduct extensive ablation studies. Notably, utilizing a transformer-based backbone, our model achieves state-of-the-art or competitive performance on five benchmarks, including PHOENIX-2014, PHOENIX-2014-T, PHOENIX-2014-SI, CSL, and CSL-Daily. Code and models are available at https://github.com/2000ZRL/LCSA_C2SLR_SRM.

Combinational sign language recognition

Boosting Continuous Sign Language Recognition via Cross Modality Augmentation

SignVTCL: Multi-Modal Continuous Sign Language Recognition Enhanced by Visual-Textual Contrastive Learning

A Sign Language Recognition Framework Based on Cross-Modal Complementary Information Fusion

Improving Continuous Sign Language Recognition with Consistency Constraints and Signer Removal

CVT-SLR: Contrastive Visual-Textual Transformation for Sign Language Recognition with Variational Alignment

Video-Based Sign Language Recognition Without Temporal Segmentation

Contrastive Learning for Sign Language Recognition and Translation.

Towards Online Continuous Sign Language Recognition and Translation

EvSign: Sign Language Recognition and Translation with Streaming Events

A Chinese Continuous Sign Language Dataset Based on Complex Environments

Improving Continuous Sign Language Recognition with Adapted Image Models

A Signer-Independent Sign Language Recognition Method for the Single-Frequency Dataset

Temporal superimposed crossover module for effective continuous sign language

Improving End-to-end Sign Language Translation with Adaptive Video Representation Enhanced Transformer

Natural Language-Assisted Sign Language Recognition

Enhancing Signer-Independent Recognition of Isolated Sign Language through Advanced Deep Learning Techniques and Feature Fusion

Collaborative Multilingual Continuous Sign Language Recognition: A Unified Framework

Sign Language Recognition with Long Short-Term Memory.

Chinese Sign Language Recognition with Sequence to Sequence Learning.