Abstract:Skeleton-based action recognition has garnered significant attention due to the utilization of concise and resilient skeletons. Nevertheless, the absence of detailed body information in skeletons restricts performance, while other multimodal methods require substantial inference resources and are inefficient when using multimodal data during both training and inference stages. To address this and fully harness the complementary multimodal features, we propose a novel multi-modality co-learning (MMCL) framework by leveraging the multimodal large language models (LLMs) as auxiliary networks for efficient skeleton-based action recognition, which engages in multi-modality co-learning during the training stage and keeps efficiency by employing only concise skeletons in inference. Our MMCL framework primarily consists of two modules. First, the Feature Alignment Module (FAM) extracts rich RGB features from video frames and aligns them with global skeleton features via contrastive learning. Second, the Feature Refinement Module (FRM) uses RGB images with temporal information and text instruction to generate instructive features based on the powerful generalization of multimodal LLMs. These instructive text features will further refine the classification scores and the refined scores will enhance the model's robustness and generalization in a manner similar to soft labels. Extensive experiments on NTU RGB+D, NTU RGB+D 120 and Northwestern-UCLA benchmarks consistently verify the effectiveness of our MMCL, which outperforms the existing skeleton-based action recognition methods. Meanwhile, experiments on UTD-MHAD and SYSU-Action datasets demonstrate the commendable generalization of our MMCL in zero-shot and domain-adaptive action recognition. Our code is publicly available at: <a class="link-external link-https" href="https://github.com/liujf69/MMCL-Action" rel="external noopener nofollow">this https URL</a>.

Multimodal Multipart Learning for Action Recognition in Depth Videos

Multimodal Multipart Learning for Action Recognition in Depth Videos

Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos

Latent Max-Margin Multitask Learning with Skelets for 3-D Action Recognition

Action Recognition by Joint Learning

Temporal Cues Enhanced Multimodal Learning for Action Recognition in RGB-D Videos

Skeleton-Indexed Deep Multi-Modal Feature Learning for High Performance Human Action Recognition

Learning Weighted Joint-based Features for Action Recognition Using Depth Camera.

Online Robust Action Recognition Based on a Hierarchical Model

Spatiotemporal Multimodal Learning With 3D CNNs for Video Action Recognition

Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition

Multimodal human action recognition based on spatio-temporal action representation recognition model

Action Recognition Using Form and Motion Modalities

Joint-based Multi-Task Sparse Learning for Human Action Recognition.

Multisource Learning for Skeleton-Based Action Recognition Using Deep LSTM and CNN

CNN-BASED ACTION RECOGNITION USING ADAPTIVE MULTISCALE DEPTH MOTION MAPS AND STABLE JOINT DISTANCE MAPS

Multi-Modal Multi-Action Video Recognition.

Multi-view key information representation and multi-modal fusion for single-subject routine action recognition

Action Recognition By Learning Deep Multi-Granular Spatio-Temporal Video Representation

Keypoints-based Multimodal Network for Robust Human Action Recognition

A Hierarchical Model For Action Recognition Based On Body Parts