Abstract:Continuous sign language recognition (CSLR) aims to identify a sequence of glosses from a sign language video with only a sentence-level label provided in a weakly supervised way. In sign language videos, the transitions among actions are naturally fluent, and different glosses or the same gloss correspond to video clips with various temporal scales. Obviously, these factors pose a challenge to the effective extraction of complex temporal information. However, most previous deep learning-based CSLR methods employ a temporal modeling method with a fixed temporal receptive field, which is a simple and effective solution but does not cope well with video clips that have various temporal scales. To relieve this problem, we propose a dual-stage temporal perception module (DTPM) by leveraging the strengths of both temporal convolutions and transformers, which follows a hierarchical structure with dual stages aimed at capturing richer and more comprehensive temporal features. Specifically, each stage for DTPM is cleverly composed of two parts: a multi-scale local temporal module (MS-LTM), followed by a set of global–local temporal modules (GLTMs), where each GLTM can be further decomposed into a global temporal relational module (GTRM) and a local temporal relational module (LTRM). At each stage, an MS-LTM is first employed to model multi-scale local temporal relations and then utilize a set of GLTMs to model global temporal relations and strengthen local temporal relations. We finally aggregate the output features of each stage to form a video feature representation with rich semantic information. Extensive experiments on three CSLR benchmarks, PHOENIX14 (Koller et al. Comput Vis Image Underst 141:108–125, 2015), PHOENIX14-T (Camgoz et al., in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7784–7793, 2018), and CSL (Huang et al., in: Proceedings of the AAAI conference on artificial intelligence, pp 32, 2018), validate the effectiveness of our proposed method.

Dynamic Pseudo Label Decoding For Continuous Sign Language Recognition

Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization.

Dilated Convolutional Network with Iterative Optimization for Continuous Sign Language Recognition

Multi-View Spatial-Temporal Network for Continuous Sign Language Recognition

Continuous Sign Language Recognition Using Multiple Feature Points

Iterative Alignment Network for Continuous Sign Language Recognition

A Deep Neural Framework for Continuous Sign Language Recognition by Iterative Training

Continuous Sign Language Recognition Using Level Building Based on Fast Hidden Markov Model

Prior-aware Cross Modality Augmentation Learning for Continuous Sign Language Recognition

Video-Based Sign Language Recognition Without Temporal Segmentation

Towards Online Continuous Sign Language Recognition and Translation

Natural Language-Assisted Sign Language Recognition

Improving Continuous Sign Language Recognition with Adapted Image Models

Deep Grammatical Multi-classifier for Continuous Sign Language Recognition.

Boosting Continuous Sign Language Recognition via Cross Modality Augmentation

Continuous Sign Language Recognition Based on Motor attention mechanism and frame-level Self-distillation

Temporal superimposed crossover module for effective continuous sign language

A Real-Time Large Vocabulary Continuous Recognition System For Chinese Sign Language

A SRN/HMM System for Signer-Independent Continuous Sign Language Recognition

A new system for Chinese sign language recognition

Dual-stage temporal perception network for continuous sign language recognition