Abstract:Sign language, which conveys meaning through gestures, is the chief means of communication among deaf people. Recognizing sign language in natural settings presents significant challenges due to factors such as lighting, background clutter, and variations in signer characteristics. In this thesis, I study automatic sign language processing in the wild, using signing videos collected from the Internet. This thesis contributes new datasets, tasks, and methods. Most chapters of this thesis address tasks related to fingerspelling, an important component of sign language and yet has not been studied widely by prior work. I present three new large-scale ASL datasets in the wild: ChicagoFSWild, ChicagoFSWild+, and OpenASL. Using ChicagoFSWild and ChicagoFSWild+, I address fingerspelling recognition, which consists of transcribing fingerspelling sequences into text. I propose an end-to-end approach based on iterative attention that allows recognition from a raw video without explicit hand detection. I further show that using a Conformer-based network jointly modeling handshape and mouthing can bring performance close to that of humans. Next, I propose two tasks for building real-world fingerspelling-based applications: fingerspelling detection and search. For fingerspelling detection, I introduce a suite of evaluation metrics and a new detection model via multi-task training. To address the problem of searching for fingerspelled keywords in raw sign language videos, we propose a novel method that jointly localizes and matches fingerspelling segments to text. Finally, I will describe a benchmark for large-vocabulary open-domain sign language translation based on OpenASL. To address the challenges of sign language translation in realistic settings, we propose a set of techniques including sign search as a pretext task for pre-training and fusion of mouthing and handshape features.

LOOKing for multi-word expressions in American Sign Language

Detecting Unseen Multiword Expressions in American Sign Language

Sign Language Video Retrieval with Free-Form Textual Queries

New Capability to Look Up an ASL Sign from a Video Example

American Sign Language Handshapes Reflect Pressures for Communicative Efficiency

Models of Language and Multiword Expressions

Analysis of the visual spatiotemporal properties of American Sign Language

Challenges for Linguistically-Driven Computer-Based Sign Recognition from Continuous Signing for American Sign Language

Collocational frequency and context effects on idiom processing in advanced L2 speakers.

Searching for fingerspelled content in American Sign Language

Recent lexical expansion in New Zealand Sign Language: context, scope and mechanisms

Cross Wavelength Comparison of Images and Catalogs

Multimodal displays of understanding in vocabulary-oriented sequences

Lookine: Let the Blind Hear a Smile.

Effort Analysis in Signer-Independent Sign Gestures.

Language as Description, Indication, and Depiction

Angry lions and scared neighbors: Complex demonstrations in sign language role shift at the sign-gesture interface

Watch, read and lookup: learning to spot signs from multiple supervisors

Repetition Preferences in Two-Handed Balanced Signs: Vestigial Locomotor Central Pattern Generators Shape Sign Language Phonetics and Phonology

ASL Video Corpora & Sign Bank: Resources Available through the American Sign Language Linguistic Research Project (ASLLRP)

Toward American Sign Language Processing in the Real World: Data, Tasks, and Methods