Abstract:Human 3D skeleton-based action recognition has received increasing interest in recent years. Inspired by the excellent ability of the multi-modal model, some pioneer attempts to employ diverse modalities, i.e., skeleton and language, to construct the skeleton-language model and have shown compelling results. Yet, these attempts model the data representation as deterministic point estimation, ignoring a key issue that descriptions of similar motions are uncertain and ambiguous, which brings about restricted comprehension of complex concept hierarchies and impoverished cross-modal alignment reliability. To tackle this challenge, this paper proposes a new Uncertain Skeleton-Language Learning Framework (USLLF) to capture the semantic ambiguity among diverse modalities in a probabilistic manner for the first time. USLLF consists of both inter- and intra-modal uncertainties. Specifically, first, we integrate the language (text) generated by ChatGPT with the generic skeleton-based network and develop a deterministic multi-modal baseline, which can be easily achieved via any off-the-shelf skeleton and text encoders. Then, based on this baseline, we explicitly model the intra-modal (skeleton/language) uncertainties as the Gaussian distributions using the new uncertainty networks capable of learning the distributional embeddings of modalities. Following this, these embeddings are aligned and formulated as inter-modal (skeleton-language) uncertainty using both the contrastive and negative log-likelihood objectives to alleviate the cross-modal alignment error. Experimental results on NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets show that our approach outperforms the proposed baseline and achieves comparable performance with a high inference efficiency compared to the state-of-the-art methods. Besides, we also deliver insightful analyses on how learned uncertainty reduces the impact of uncertain and ambiguous data on model performance.

SKIM: Skeleton-Based Isolated Sign Language Recognition With Part Mixing

Skeleton Aware Multi-modal Sign Language Recognition

Sign Language Recognition via Skeleton-Aware Multi-Model Ensemble

Attention-Based Sign Language Recognition Network Utilizing Keyframe Sampling and Skeletal Features

Dynamic Spatial-Temporal Aggregation for Skeleton-Aware Sign Language Recognition

Sign Language Recognition Based on Skeleton and SK3D-Residual Network

Multimodal Fusion Based on LSTM and a Couple Conditional Hidden Markov Model for Chinese Sign Language Recognition

Sign Language Recognition with Multi-modal Features.

Skeleton-Aware Neural Sign Language Translation.

TMS-Net: A multi-feature multi-stream multi-level information sharing network for skeleton-based sign language recognition

Independent Sign Language Recognition with 3D Body, Hands, and Face Reconstruction

StepNet: Spatial-temporal Part-aware Network for Isolated Sign Language Recognition

Sign Language Recognition Based On Facial Expression and Hand Skeleton

Natural Language-Assisted Sign Language Recognition

A Signer-Independent Sign Language Recognition Method for the Single-Frequency Dataset

Interactive attention and improved GCN for continuous sign language recognition

Deep Forest-Based Monocular Visual Sign Language Recognition

Modeling the Skeleton-Language Uncertainty for 3D Action Recognition

Hand-Model-Aware Sign Language Recognition

Enhancing Brazilian Sign Language Recognition through Skeleton Image Representation