Abstract:Although sign language recognition aids non-hearing-impaired understanding, many hearing-impaired individuals still rely on sign language alone due to limited literacy, underscoring the need for advanced sign language production and translation (SLP and SLT) systems. In the field of sign language production, the lack of adequate models and datasets restricts practical applications. Existing models face challenges in production accuracy and pose control, making it difficult to provide fluent sign language expressions across diverse scenarios. Additionally, data resources are scarce, particularly high-quality datasets with complete sign vocabulary and pose annotations. To address these issues, we introduce CNText2Sign and CNSign, comprehensive datasets to benchmark SLP and SLT, respectively, with CNText2Sign covering gloss and landmark mappings for SLP, and CNSign providing extensive video-to-text data for SLT. To improve the accuracy and applicability of sign language systems, we propose the AuraLLM and SignMST-C models. AuraLLM, incorporating LoRA and RAG techniques, achieves a BLEU-4 score of 50.41 on the CNText2Sign dataset, enabling precise control over gesture semantics and motion. SignMST-C employs self-supervised rapid motion video pretraining, achieving a BLEU-4 score of 31.03/32.08 on the PHOENIX2014-T benchmark, setting a new state-of-the-art. These models establish robust baselines for the datasets released for their respective tasks.

What problem does this paper attempt to address?

The main problems that this paper attempts to solve are the deficiencies in the accuracy and fluency of existing Sign Language Production (SLP) and Sign Language Translation (SLT) systems, especially when dealing with complex scenarios and diverse semantic expressions. Specifically: 1. **Challenges of Sign Language Production (SLP) Systems**: - Existing SLP models face challenges in generation accuracy and pose control, and it is difficult to provide fluent sign language expressions. - There is a lack of high - quality datasets, especially high - quality datasets that contain complete sign language vocabularies and pose annotations. 2. **Challenges of Sign Language Translation (SLT) Systems**: - Existing SLT methods are difficult to precisely align dynamic visual gestures and language semantics, resulting in difficulty in capturing subtle movement details and generating grammatically correct and semantically coherent texts. - SLP and SLT are usually studied as independent tasks, lacking an integrated design to solve the inter - related problems in practical applications. To solve these problems, the author proposes the following solutions: 1. **Introducing New Datasets**: - **CNText2Sign**: This is a large - scale Chinese natural language - to - sign - language - vocabulary mapping dataset, which contains 15,000 annotated mappings and 8,643 officially defined core Chinese sign - language vocabulary items, and each vocabulary item is accompanied by corresponding video data. These data extract skeleton key points through MediaPipe to generate complete pose sequences, ensuring semantic consistency. - **CNSign**: This is a dataset for Chinese sign - language translation, which contains 10,943 video - to - text mappings. The video duration ranges from 5 seconds to 32 seconds, covering complex syntactic structures and being suitable for various practical application scenarios. 2. **Proposing New Models**: - **AuraLLM**: This is a sign - language - production model that combines LoRA (Low - Rank Adaptation) and RAG (Retrieval - Augmented Generation) techniques. It achieves a BLEU - 4 score of 50.41% on the CNText2Sign dataset and can precisely control gesture semantics and movements. - **SignMST - C**: This is a sign - language - translation model based on self - supervised pre - training and multi - modal feature fusion. It achieves a BLEU - 4 score of 31.03/32.08 on the PHOENIX2014 - T benchmark, setting a new state - of - the - art level. Through these improvements, the paper aims to improve the accuracy and applicability of sign - language - production and - translation systems, thereby better meeting the needs of the hearing - impaired and promoting the development of barrier - free communication.

Beyond Words: AuralLLM and SignMST-C for Precise Sign Language Production and Bidirectional Accessibility

SignLLM: Sign Language Production Large Language Models

SignVTCL: Multi-Modal Continuous Sign Language Recognition Enhanced by Visual-Textual Contrastive Learning

Natural Language-Assisted Sign Language Recognition

MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognition

BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues

Improving Continuous Sign Language Recognition with Adapted Image Models

SCOPE: Sign Language Contextual Processing with Embedding from LLMs

Event Stream based Sign Language Translation: A High-Definition Benchmark Dataset and A New Algorithm

MLSLT: Towards Multilingual Sign Language Translation.

An Improved Sign Language Translation Model with Explainable Adaptations for Processing Long Sign Sentences

Improving Continuous Sign Language Recognition with Consistency Constraints and Signer Removal

SignAvatars: A Large-scale 3D Sign Language Holistic Motion Dataset and Benchmark

Fine-tuning of sign language recognition models: a technical report

Improving Continuous Sign Language Recognition with Cross-Lingual Signs

SignBLEU: Automatic Evaluation of Multi-channel Sign Language Translation

StepNet: Spatial-temporal Part-aware Network for Isolated Sign Language Recognition

A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars

CSLNSpeech: solving the extended speech separation problem with the help of Chinese Sign Language

Sign Language Recognition with Long Short-Term Memory.