Abstract:Since American Sign Language (ASL) has no standard written form, Deaf signers frequently share videos in order to communicate in their native language. However, since both hands and face convey critical linguistic information in signed languages, sign language videos cannot preserve signer privacy. While signers have expressed interest, for a variety of applications, in sign language video anonymization that would effectively preserve linguistic content, attempts to develop such technology have had limited success, given the complexity of hand movements and facial expressions. Existing approaches rely predominantly on precise pose estimations of the signer in video footage and often require sign language video datasets for training. These requirements prevent them from processing videos 'in the wild,' in part because of the limited diversity present in current sign language video datasets. To address these limitations, our research introduces DiffSLVA, a novel methodology that utilizes pre-trained large-scale diffusion models for zero-shot text-guided sign language video anonymization. We incorporate ControlNet, which leverages low-level image features such as HED (Holistically-Nested Edge Detection) edges, to circumvent the need for pose estimation. Additionally, we develop a specialized module dedicated to capturing facial expressions, which are critical for conveying essential linguistic information in signed languages. We then combine the above methods to achieve anonymization that better preserves the essential linguistic content of the original signer. This innovative methodology makes possible, for the first time, sign language video anonymization that could be used for real-world applications, which would offer significant benefits to the Deaf and Hard-of-Hearing communities. We demonstrate the effectiveness of our approach with a series of signer anonymization experiments.

FLEURS-ASL: Including American Sign Language in Massively Multilingual Multitask Evaluation

Sign Language Video Retrieval with Free-Form Textual Queries

YouTube-SL-25: A Large-Scale, Open-Domain Multilingual Sign Language Parallel Corpus

DeepASL: Enabling Ubiquitous and Non-Intrusive Word and Sentence-Level Sign Language Translation

Scaling Sign Language Translation

YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpus

SignMusketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scale

American Sign Language Translation Using Wearable Inertial and Electromyography Sensors for Tracking Hand Movements and Facial Expressions

Reconsidering Sentence-Level Sign Language Translation

Two-Stream Network for Sign Language Recognition and Translation

SignBLEU: Automatic Evaluation of Multi-channel Sign Language Translation

Fingerspelling within Sign Language Translation

Gloss-Free End-to-End Sign Language Translation

Empowering Sign Language Communication: Integrating Sentiment and Semantics for Facial Expression Synthesis

Event Stream based Sign Language Translation: A High-Definition Benchmark Dataset and A New Algorithm

A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars

Natural Language-Assisted Sign Language Recognition

Toward American Sign Language Processing in the Real World: Data, Tasks, and Methods

Improving Massively Multilingual ASR With Auxiliary CTC Objectives

DiffSLVA: Harnessing Diffusion Models for Sign Language Video Anonymization

A Classification Model Utilizing Facial Landmark Tracking to Determine Sentence Types for American Sign Language Recognition