TDSM: Triplet Diffusion for Skeleton-Text Matching in Zero-Shot Action Recognition

Jeonghyeok Do,Munchurl Kim

2024-11-22

Abstract:We firstly present a diffusion-based action recognition with zero-shot learning for skeleton inputs. In zero-shot skeleton-based action recognition, aligning skeleton features with the text features of action labels is essential for accurately predicting unseen actions. Previous methods focus on direct alignment between skeleton and text latent spaces, but the modality gaps between these spaces hinder robust generalization learning. Motivated from the remarkable performance of text-to-image diffusion models, we leverage their alignment capabilities between different modalities mostly by focusing on the training process during reverse diffusion rather than using their generative power. Based on this, our framework is designed as a Triplet Diffusion for Skeleton-Text Matching (TDSM) method which aligns skeleton features with text prompts through reverse diffusion, embedding the prompts into the unified skeleton-text latent space to achieve robust matching. To enhance discriminative power, we introduce a novel triplet diffusion (TD) loss that encourages our TDSM to correct skeleton-text matches while pushing apart incorrect ones. Our TDSM significantly outperforms the very recent state-of-the-art methods with large margins of 2.36%-point to 13.05%-point, demonstrating superior accuracy and scalability in zero-shot settings through effective skeleton-text matching.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to effectively align skeletal features and text - label features in zero - shot skeletal action recognition, so as to accurately predict unseen actions. Specifically, the paper proposes improvement schemes for the following problems:

1. **Modality Gaps**: Traditional methods directly align between the skeletal feature space and the text feature space. However, due to the inherent differences between these two modalities, it is difficult for the model to generalize to unseen actions.

2. **Effective Cross - Modal Alignment**: In order to overcome the challenges brought by modality gaps, a more effective method is required to fuse skeletal features and text descriptions to ensure robust matching in a unified latent space.

3. **Improving Discriminative Ability**: In order to improve the discriminative ability of the model, a loss function is needed that can encourage correct skeletal - text matching and push incorrect matches further apart.

To solve these problems, the paper proposes a Triplet Diffusion method for Skeleton - Text Matching (TDSM) based on the diffusion model. This method embeds text prompts into the unified skeletal - text latent space through the reverse diffusion process to achieve more effective cross - modal alignment, and introduces a new Triplet Diffusion Loss (TD loss), which further enhances the discriminative ability of the model.

### Main Contributions
1. **First Application of the Diffusion Model**: TDSM is the first method to apply the diffusion model to zero - shot skeletal action recognition, using the excellent cross - modal correspondence learning ability of the diffusion model to achieve implicit alignment between skeletal features and text prompts.

2. **Novel Triplet Diffusion Loss**: The Triplet Diffusion Loss (TD loss) is introduced, which enhances the discriminative ability of the model by ensuring close alignment of correct skeletal - text pairs and pushing incorrect pairs apart.

3. **Significantly Outperforms Existing Methods**: TDSM significantly outperforms the latest SOTA methods on multiple benchmark datasets, with a performance improvement of 2.36% to 13.05%, demonstrating its scalability and robustness in various scenarios.

Through these improvements, TDSM can more accurately identify unseen actions in a zero - shot setting, solving the shortcomings of traditional methods in terms of modality gaps and generalization ability.