Abstract:Skeleton-based action recognition has garnered significant attention due to the utilization of concise and resilient skeletons. Nevertheless, the absence of detailed body information in skeletons restricts performance, while other multimodal methods require substantial inference resources and are inefficient when using multimodal data during both training and inference stages. To address this and fully harness the complementary multimodal features, we propose a novel multi-modality co-learning (MMCL) framework by leveraging the multimodal large language models (LLMs) as auxiliary networks for efficient skeleton-based action recognition, which engages in multi-modality co-learning during the training stage and keeps efficiency by employing only concise skeletons in inference. Our MMCL framework primarily consists of two modules. First, the Feature Alignment Module (FAM) extracts rich RGB features from video frames and aligns them with global skeleton features via contrastive learning. Second, the Feature Refinement Module (FRM) uses RGB images with temporal information and text instruction to generate instructive features based on the powerful generalization of multimodal LLMs. These instructive text features will further refine the classification scores and the refined scores will enhance the model's robustness and generalization in a manner similar to soft labels. Extensive experiments on NTU RGB+D, NTU RGB+D 120 and Northwestern-UCLA benchmarks consistently verify the effectiveness of our MMCL, which outperforms the existing skeleton-based action recognition methods. Meanwhile, experiments on UTD-MHAD and SYSU-Action datasets demonstrate the commendable generalization of our MMCL in zero-shot and domain-adaptive action recognition. Our code is publicly available at: <a class="link-external link-https" href="https://github.com/liujf69/MMCL-Action" rel="external noopener nofollow">this https URL</a>.

Advancing Human Motion Recognition with SkeletonCLIP++: Weighted Video Feature Integration and Enhanced Contrastive Sample Discrimination

SkeletonCLIP: Recognizing Skeleton-based Human Actions with Text Prompts

Shifting Perspective to See Difference: A Novel Multi-View Method for Skeleton Based Action Recognition

Skeleton edge motion networks for human action recognition

A Skeleton-based Action Recognition System for Medical Condition Detection

Improved skeleton-based activity recognition using convolutional block attention module

Wavelet-Decoupling Contrastive Enhancement Network for Fine-Grained Skeleton-Based Action Recognition

Skeleton Focused Human Activity Recognition in RGB Video

Skeleton-Indexed Deep Multi-Modal Feature Learning for High Performance Human Action Recognition

Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep cnn

Enhanced Skeleton Visualization for View Invariant Human Action Recognition.

Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition

Enhancing Skeleton-Based Action Recognition with Language Descriptions from Pre-trained Large Multimodal Models

Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition

Skeleton-based action recognition with hierarchical spatial reasoning and temporal stack learning network

Learning Discriminative Representations for Skeleton Based Action Recognition

Human posture estimation and action recognition on fitness behavior and fitness

Motion Complement and Temporal Multifocusing for Skeleton-Based Action Recognition

Shap-Mix: Shapley Value Guided Mixing for Long-Tailed Skeleton Based Action Recognition

An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition

Skeleton-Based Mutually Assisted Interacted Object Localization and Human Action Recognition