Abstract:Search Engines such as Google, Baidu, and Bing have revolutionized the way we interact with the cyber world with a number of applications in recommendations, learning, advertisements, healthcare, entertainment, etc. In this paper, we design search engines for sign languages such as American Sign Language (ASL). Sign languages use hand and body motion for communication with rich grammar, complexity, and vocabulary that is comparable to spoken languages. This is the primary language for the Deaf community with a global population of ≈ 500 million. However, search engines that support sign language queries in native form do not exist currently. While translating a sign language to a spoken language and using existing search engines might be one possibility, this can miss critical information because existing translation systems are either limited in vocabulary or constrained to a specific domain. In contrast, this paper presents a holistic approach where ASL queries in native form as well as ASL videos and textual information available online are converted into a common representation space. Such a joint representation space provides a common framework for precisely representing different sources of information and accurately matching a query with relevant information that is available online. Our system uses low-intrusive wearable sensors for capturing the sign query. To minimize the training overhead, we obtain synthetic training data from a large corpus of online ASL videos across diverse topics. Evaluated over a set of Deaf users with native ASL fluency, the accuracy is comparable with state-of-the-art recommendation systems for Amazon, Netflix, Yelp, etc., suggesting the usability of the system in the real world. For example, the re-call@10 of our system is 64.3%, i.e., among the top ten search results, six of them are relevant to the search query. Moreover, the system is robust to variations in signing patterns, dialects, sensor positions, etc.

Demo: The Sound of Silence: End-to-End Sign Language Recognition Using SmartWatch.

SignSpeaker: A Real-time, High-Precision SmartWatch-based Sign Language Translator

Hear Sign Language: A Real-Time End-to-End Sign Language Recognition System

MyoSign

MyoSign: enabling end-to-end sign language recognition with wearables

A wearable system for sign language recognition enabled by a convolutional neural network

HearASL: Your Smartphone Can Hear American Sign Language

American Sign Language Translation Using Wearable Inertial and Electromyography Sensors for Tracking Hand Movements and Facial Expressions

American Sign Language Recognition and Translation Using Perception Neuron Wearable Inertial Motion Capture System

SmartASL

Sign Language Conversation Interpretation Using Wearable Sensors and Machine Learning

TransASL: A Smart Glass Based Comprehensive ASL Recognizer in Daily Life.

WearSign: Pushing the Limit of Sign Language Translation Using Inertial and EMG Wearables.

SignQuery: A Natural User Interface and Search Engine for Sign Languages with Wearable Sensors

Automatic recognition of sign language subwords based on portable accelerometer and EMG sensors

End-to-End Two-Handed Sign Language Translation

An Approach Based on 1D Fully Convolutional Network for Continuous Sign Language Recognition and Labeling

An EMG Based Wearable System for Chinese Sign Language Recognition

Sign language recognition system for communicating to people with disabilities

SonicASL

L-Sign: Large-Vocabulary Sign Gestures Recognition System