Abstract:Skeleton-based action recognition has garnered significant attention due to the utilization of concise and resilient skeletons. Nevertheless, the absence of detailed body information in skeletons restricts performance, while other multimodal methods require substantial inference resources and are inefficient when using multimodal data during both training and inference stages. To address this and fully harness the complementary multimodal features, we propose a novel multi-modality co-learning (MMCL) framework by leveraging the multimodal large language models (LLMs) as auxiliary networks for efficient skeleton-based action recognition, which engages in multi-modality co-learning during the training stage and keeps efficiency by employing only concise skeletons in inference. Our MMCL framework primarily consists of two modules. First, the Feature Alignment Module (FAM) extracts rich RGB features from video frames and aligns them with global skeleton features via contrastive learning. Second, the Feature Refinement Module (FRM) uses RGB images with temporal information and text instruction to generate instructive features based on the powerful generalization of multimodal LLMs. These instructive text features will further refine the classification scores and the refined scores will enhance the model's robustness and generalization in a manner similar to soft labels. Extensive experiments on NTU RGB+D, NTU RGB+D 120 and Northwestern-UCLA benchmarks consistently verify the effectiveness of our MMCL, which outperforms the existing skeleton-based action recognition methods. Meanwhile, experiments on UTD-MHAD and SYSU-Action datasets demonstrate the commendable generalization of our MMCL in zero-shot and domain-adaptive action recognition. Our code is publicly available at: <a class="link-external link-https" href="https://github.com/liujf69/MMCL-Action" rel="external noopener nofollow">this https URL</a>.

DMCL: Distillation Multiple Choice Learning for Multimodal Action Recognition

Modality Distillation with Multiple Stream Networks for Action Recognition

Cross-modality Online Distillation for Multi-View Action Recognition

CMD: Self-supervised 3D Action Representation Learning with Cross-Modal Mutual Distillation

Multimodal Distillation for Egocentric Action Recognition

Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition

One-stage Modality Distillation for Incomplete Multimodal Learning

Multi-modal learning for gesture recognition

Modality Mixer Exploiting Complementary Information for Multi-modal Action Recognition

I$^2$MD: 3D Action Representation Learning with Inter- and Intra-modal Mutual Distillation

Dense Adaptive Grouping Distillation Network for Multimodal Land Cover Classification with Privileged Modality

Modality Compensation Network: Cross-Modal Adaptation for Action Recognition

SMTDKD: A Semantic-Aware Multimodal Transformer Fusion Decoupled Knowledge Distillation Method for Action Recognition

Multi-view Distillation based on Multi-modal Fusion for Few-shot Action Recognition(CLIP-$\mathrm{M^2}$DF)

DFN: A deep fusion network for flexible single and multi-modal action recognition

A Bidirectional Separated Distillation-Based Cross-Modal Interactive Fusion Network for Skeleton-Based Action Recognition

Dealing with All-stage Missing Modality: Towards A Universal Model with Robust Reconstruction and Personalization

Implicit Attention-Based Cross-Modal Collaborative Learning for Action Recognition

Multi-Modal Multi-Action Video Recognition.

Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection

Multimodal Multipart Learning for Action Recognition in Depth Videos