Abstract:Sign language recognition (SLR) has long been plagued by insufficient model representation capabilities. Although current pre-training approaches have alleviated this dilemma to some extent and yielded promising performance by employing various pretext tasks on sign pose data, these methods still suffer from two primary limitations: 1) Explicit motion information is usually disregarded in previous pretext tasks, leading to partial information loss and limited representation capability. 2) Previous methods focus on the local context of a sign pose sequence, without incorporating the guidance of the global meaning of lexical signs. To this end, we propose a Motion-Aware masked autoencoder with Semantic Alignment (MASA) that integrates rich motion cues and global semantic information in a self-supervised learning paradigm for SLR. Our framework contains two crucial components, i.e., a motion-aware masked autoencoder (MA) and a momentum semantic alignment module (SA). Specifically, in MA, we introduce an autoencoder architecture with a motion-aware masked strategy to reconstruct motion residuals of masked frames, thereby explicitly exploring dynamic motion cues among sign pose sequences. Moreover, in SA, we embed our framework with global semantic awareness by aligning the embeddings of different augmented samples from the input sequence in the shared latent space. In this way, our framework can simultaneously learn local motion cues and global semantic features for comprehensive sign language representation. Furthermore, we conduct extensive experiments to validate the effectiveness of our method, achieving new state-of-the-art performance on four public benchmarks.

Combining PSM and HMM for unsupervised signer adaptation

Combining HMM and SPSM for Sign Language Recognition

Generating data for signer adaptation

Mahalanobis Distance Based Polynomial Segment Model for Chinese Sign Language Recogniton

Sign Language Recognition Based on Adaptive Hmms with Data Augmentation

Unsupervised Adaptive Sign Language Recognition Based on Hypothesis Comparison Guided Cross Validation and Linguistic Prior Filtering

Chinese sign language recognition with adaptive HMM

Vision-Based Sign Language Recognition Using Sign-Wise Tied Mixture Hmm

Signer-Independent Sign Language Recognition Based On Sofm/Hmm

A Sofm/Hmm System for Person-Independent Isolated Sign Language Recognition

Online Early-Late Fusion Based on Adaptive HMM for Sign Language Recognition

Recognition of Sign Language Subwords Based on Boosted Hidden Markov Models.

MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognition

Adaptive Sign Language Recognition with Exemplar Extraction and MAP/IVFS

A Chinese Sign Language Recognition System Based on SOFM/SRN/HMM

A SRN/HMM System for Signer-Independent Continuous Sign Language Recognition

A Vision-Based Sign Language Recognition System Using Tied-Mixture Density HMM

A Signer-Independent Sign Language Recognition Method for the Single-Frequency Dataset

THE APPLICATION OF HMMs CONSISTING OF DIFFERENT NUMBER STATES IN CHINESE SIGN LANGUAGE RECOGNITION

Signer-Independent Continuous Sign Language Recognition Based on SRN/HMM

An adaptive Chinese sign language recognition method based on data generating