Diffusion-based Human Motion Style Transfer with Semantic Guidance

Lei Hu,Zihao Zhang,Yongjing Ye,Yiwen Xu,Shihong Xia

2024-08-07

Abstract:3D Human motion style transfer is a fundamental problem in computer graphic and animation processing. Existing AdaIN- based methods necessitate datasets with balanced style distribution and content/style labels to train the clustered latent space. However, we may encounter a single unseen style example in practical scenarios, but not in sufficient quantity to constitute a style cluster for AdaIN-based methods. Therefore, in this paper, we propose a novel two-stage framework for few-shot style transfer learning based on the diffusion model. Specifically, in the first stage, we pre-train a diffusion-based text-to-motion model as a generative prior so that it can cope with various content motion inputs. In the second stage, based on the single style example, we fine-tune the pre-trained diffusion model in a few-shot manner to make it capable of style transfer. The key idea is regarding the reverse process of diffusion as a motion-style translation process since the motion styles can be viewed as special motion variations. During the fine-tuning for style transfer, a simple yet effective semantic-guided style transfer loss coordinated with style example reconstruction loss is introduced to supervise the style transfer in CLIP semantic space. The qualitative and quantitative evaluations demonstrate that our method can achieve state-of-the-art performance and has practical applications.

Graphics,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to obtain a model capable of converting various content actions into a given style through few - shot learning when only one unseen style example is provided. Specifically, the existing AdaIN - based methods require a data set with a balanced style distribution and content / style labels to train the clustered latent space. However, in real - world scenarios, we may only have one or a small number of style examples, which are not sufficient to form the style clusters required by the AdaIN method. Therefore, this paper proposes a two - stage framework based on the diffusion model for few - sample style transfer learning.

### Main Contributions
1. **Novel Style Transfer Framework**: Consider the inverse process of the diffusion model as a style translation process.
2. **Semantically - Guided Style Transfer Learning Strategy**: It can supervise human action style transfer in the CLIP semantic space.
3. **Extensive Experimental Verification**: It is proved that the proposed framework can achieve unseen style transfer by fine - tuning using only one style example.

### Method Overview
1. **Pre - training Stage**:
- Pre - train a text - to - action generation model (εθ) based on the diffusion model as a generation prior.
- Pre - train an action - semantic discriminator (Dis) to measure the similarity between actions and text prompts.

2. **Fine - tuning Stage**:
- Given a style example (ms) and its content and style prompts (ds), obtain the style - transfer diffusion model (εθs) related to ms through fine - tuning.
- Introduce the style example reconstruction loss (Lsr) and the CLIP - based semantically - guided learning loss (Ls) to jointly fine - tune the network.

3. **Inference Stage**:
- Use the fine - tuned style - transfer diffusion model (εθs) to apply the style ms to various content actions mc.
- If the source action mc is not style - neutral, first use the generation prior εθ to generate the corresponding neutral action as an intermediate state, and then achieve style transfer through the forward process (mc→mcK) and the reverse process (mcK→mc_s).

### Key Technologies
- **Diffusion Model**: Add noise through the forward diffusion process and restore actions through the reverse denoising process while introducing style changes.
- **Semantic Guidance**: Utilize the pre - trained action - semantic discriminator and the CLIP text encoder to ensure that the generated actions are semantically consistent with the target style.
- **Style Example Reconstruction**: Learn style differences by generating neutral actions and comparing them with style examples.

### Formulas
- **Forward Diffusion Process**:
\[
q(m_t | m_{t - 1})=\mathcal{N}(\sqrt{\alpha_t}m_{t - 1},(1 - \alpha_t)I)
\]
where \( q \) is the conditional probability distribution, \( \alpha_t\in(0, 1) \) is a constant hyperparameter, and \( I \) is the identity matrix.

- **Reverse Denoising Process**:
\[
\hat{m}_0(t)=\epsilon_\theta(m_t, t, c)
\]
where \( c \) is the conditional signal and \( \hat{m}_0(t) \) is the initial action predicted from \( m_t \) at time step \( t \).

- **Simple Loss Function**:
\[
L_{\text{simple}}=\mathbb{E}_{m, t\sim[1, T]}\left[\|m_0 - \epsilon_\theta(m_t, t, d)\|^2_2\right]
\]

- **Semantic - Guided Loss**:
\[
L_s=\mathbb{E}_{m_c^s, d_c^s}\left[1 - \frac{\text{Dis}(m_c^s)\cdot\text{Etext}(d_c^s)}{\max(\|\text{Dis}(m_c^s)\|\times\|\text{Etex}