Abstract:End-to-end sign language translation (SLT) aims to convert sign language videos into spoken language texts directly without intermediate representations. It has been a challenging task due to the modality gap between sign videos and texts and the data scarcity of labeled data. Due to these challenges, the input and output distributions of end-to-end sign language translation (i.e., video-to-text) are less effective compared to the gloss-to-text approach (i.e., text-to-text). To tackle these challenges, we propose a novel Cross-modality Data Augmentation (XmDA) framework to transfer the powerful gloss-to-text translation capabilities to end-to-end sign language translation (i.e. video-to-text) by exploiting pseudo gloss-text pairs from the sign gloss translation model. Specifically, XmDA consists of two key components, namely, cross-modality mix-up and cross-modality knowledge distillation. The former explicitly encourages the alignment between sign video features and gloss embeddings to bridge the modality gap. The latter utilizes the generation knowledge from gloss-to-text teacher models to guide the spoken language text generation. Experimental results on two widely used SLT datasets, i.e., PHOENIX-2014T and CSL-Daily, demonstrate that the proposed XmDA framework significantly and consistently outperforms the baseline models. Extensive analyses confirm our claim that XmDA enhances spoken language text generation by reducing the representation distance between videos and texts, as well as improving the processing of low-frequency words and long sentences.

Self-Augmentation Improves Zero-Shot Cross-Lingual Transfer

A Simple and Effective Method to Improve Zero-Shot Cross-Lingual Transfer Learning.

Boosting Zero-Shot Crosslingual Performance using LLM-Based Augmentations with Effective Data Selection

CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP

Self-Translate-Train: Enhancing Cross-Lingual Transfer of Large Language Models via Inherent Capability

Improving Cross-Lingual Reading Comprehension with Self-Training

UniPSDA: Unsupervised Pseudo Semantic Data Augmentation for Zero-Shot Cross-Lingual Natural Language Understanding

Attention-Informed Mixed-Language Training for Zero-Shot Cross-Lingual Task-Oriented Dialogue Systems

Improved Cross-Lingual Transfer Learning For Automatic Speech Translation

Cross-lingual Back-Parsing: Utterance Synthesis from Meaning Representation for Zero-Resource Semantic Parsing

XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering

Boosting Cross-Lingual Transfer via Self-Learning with Uncertainty Estimation

Improving Zero-Shot Cross-Lingual Transfer via Progressive Code-Switching

Enhancing Code-Switching for Cross-lingual SLU: A Unified View of Semantic and Grammatical Coherence

Zero-Resource Multilingual Model Transfer: Learning What to Share

Cross-lingual Transfer or Machine Translation? On Data Augmentation for Monolingual Semantic Textual Similarity

Bridging Cross-Lingual Gaps During Leveraging the Multilingual Sequence-to-Sequence Pretraining for Text Generation and Understanding

Data Augmentation for Code-Switch Language Modeling by Fusing Multiple Text Generation Methods.

SDA: Improving Text Generation with Self Data Augmentation

Cross-modality Data Augmentation for End-to-End Sign Language Translation

Self-Augmented In-Context Learning for Unsupervised Word Translation