Abstract:Sign Language Translation (SLT) is a challenging task due to its cross-domain nature, involving the translation of visual-gestural language to text. Many previous methods employ an intermediate representation, i.e., gloss sequences, to facilitate SLT, thus transforming it into a two-stage task of sign language recognition (SLR) followed by sign language translation (SLT). However, the scarcity of gloss-annotated sign language data, combined with the information bottleneck in the mid-level gloss representation, has hindered the further development of the SLT task. To address this challenge, we propose a novel Gloss-Free SLT based on Visual-Language Pretraining (GFSLT-VLP), which improves SLT by inheriting language-oriented prior knowledge from pre-trained models, without any gloss annotation assistance. Our approach involves two stages: (i) integrating Contrastive Language-Image Pre-training (CLIP) with masked self-supervised learning to create pre-tasks that bridge the semantic gap between visual and textual representations and restore masked sentences, and (ii) constructing an end-to-end architecture with an encoder-decoder-like structure that inherits the parameters of the pre-trained Visual Encoder and Text Decoder from the first stage. The seamless combination of these novel designs forms a robust sign language representation and significantly improves gloss-free sign language translation. In particular, we have achieved unprecedented improvements in terms of BLEU-4 score on the PHOENIX14T dataset (>+5) and the CSL-Daily dataset (>+3) compared to state-of-the-art gloss-free SLT methods. Furthermore, our approach also achieves competitive results on the PHOENIX14T dataset when compared with most of the gloss-based methods. Our code is available at <a class="link-external link-https" href="https://github.com/zhoubenjia/GFSLT-VLP" rel="external noopener nofollow">this https URL</a>.

Gloss Semantic-Enhanced Network with Online Back-Translation for Sign Language Production

Towards Fast and High-Quality Sign Language Production

Gloss-Free End-to-End Sign Language Translation

A Novel POS-Guided Data Augmentation Method for Sign Language Gloss Translation.

Autoregressive Sign Language Production: A Gloss-Free Approach with Discrete Representations

Sign Language Production with Latent Motion Transformer

Gloss Attention for Gloss-Free Sign Language Translation

G2P-DDM: Generating Sign Pose Sequence from Gloss Sequence with Discrete Diffusion Model

Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining

Everybody Sign Now: Translating Spoken Language to Photo Realistic Sign Language Video

Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation

Improving Gloss-free Sign Language Translation by Reducing Representation Density

VK-G2T: Vision and Context Knowledge enhanced Gloss2Text

Linguistics-Vision Monotonic Consistent Network for Sign Language Production

Better Sign Language Translation with Monolingual Data

Graph Adversarial Network with Bottleneck Adapter Tuning for Sign Language Production

Gloss Prior Guided Visual Feature Learning for Continuous Sign Language Recognition

Sign2Pose: A Pose-Based Approach for Gloss Prediction Using a Transformer Model

A Spatio-Temporal Representation Learning as an Alternative to Traditional Glosses in Sign Language Translation and Production

Progressive Transformers for End-to-End Sign Language Production

Data Augmentation for Sign Language Gloss Translation