Abstract:Human motion synthesis conditioned on textual input has gained significant attention in recent years due to its potential applications in various domains such as gaming, film production, and virtual reality. Conditioned Motion synthesis takes a text input and outputs a 3D motion corresponding to the text. While previous works have explored motion synthesis using raw motion data and latent space representations with diffusion models, these approaches often suffer from high training and inference times. In this paper, we introduce a novel framework that utilizes Generative Adversarial Networks (GANs) in the latent space to enable faster training and inference while achieving results comparable to those of the state-of-the-art diffusion methods. We perform experiments on the HumanML3D, HumanAct12 benchmarks and demonstrate that a remarkably simple GAN in the latent space achieves a FID of 0.482 with more than 91% in FLOPs reduction compared to latent diffusion model. Our work opens up new possibilities for efficient and high-quality motion synthesis using latent space GANs.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to improve the efficiency and quality of text - input - based human motion synthesis. Specifically, the paper aims to: 1. **Accelerate training and inference time**: By introducing a new framework and using Generative Adversarial Networks (GANs) to perform motion synthesis in the latent space, thus reducing the time cost of training and inference. 2. **Maintain high - quality motion synthesis results**: Ensure that while accelerating, the quality of the generated motion sequences can be comparable to that of the state - of - the - art diffusion models. ### Problem Background In recent years, text - input - based human motion synthesis has attracted wide attention because of its great potential for application in fields such as games, film production, and virtual reality. However, existing methods (such as diffusion models based on original motion data and latent space representations) often have problems of high computational cost and long training and inference times. ### Solutions Proposed in the Paper To solve these problems, the paper proposes the following innovations: - **Use GANs instead of diffusion models**: Use Generative Adversarial Networks (GANs) in the latent space to take advantage of their efficient adversarial training mechanism, thereby achieving faster training and inference. - **Combine Variational Auto - Encoder (VAE) and pre - trained CLIP model**: Use VAE to transform motion sequences into the latent space and use the pre - trained CLIP model to condition the text input. - **Experiment with multiple GAN architectures**: Including simple GAN, deep GAN, and Wasserstein GAN (WGAN), and optimize the loss functions (such as cross - entropy and Wasserstein loss) to improve performance and fidelity. ### Experimental Results The paper conducted experiments on the HumanML3D and HumanAct12 benchmarks, and the results show that: - A simple GAN architecture achieved an FID (Fréchet Inception Distance) of 0.482 in the latent space and reduced more than 91% of FLOPs (floating - point operations) compared to the latent diffusion model. - In the action - to - motion synthesis task, Deep WGAN - GP outperforms other models in terms of diversity while remaining competitive in terms of FID, accuracy, and multi - modality. ### Summary By using GANs in the latent space, the method proposed in the paper not only solves the computational efficiency problem of existing diffusion models but also improves the quality and diversity of motion synthesis, which is suitable for real - time applications. This work has opened up a new direction for efficient and high - quality motion synthesis. ### Formula Presentation The formulas involved in the paper mainly include: - **Encoding and decoding processes of VAE**: \[ z = E(x_{1:L}) \] \[ \hat{x}_{1:L} = D(z) = D(E(x_{1:L})) \] - **Training objective of GAN**: \[ \min_G \max_D \mathbb{E}_{z \sim p_z(z)}[\log D(z, c)] + \mathbb{E}_{z \sim p_z(z)}[\log(1 - D(G(z, c), c))] \] These formulas show how to generate and discriminate motion sequences in the latent space, ensuring the theoretical basis and implementation details of the method.

LS-GAN: Human Motion Synthesis with Latent-space GANs

OAW-GAN: Occlusion-Aware Warping GAN for Unified Human Video Synthesis

MotionGPT: Human Motion Synthesis with Improved Diversity and Realism via GPT-3 Prompting

Autoregressive GAN for Semantic Unconditional Head Motion Generation

Aggregated Multi-GANs for Controlled 3D Human Motion Prediction

Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance

Towards Practical Single-shot Motion Synthesis

MotionGPT: Finetuned LLMs Are General-Purpose Motion Generators

Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control

GANimator: Neural Motion Synthesis from a Single Sequence

BiHMP-GAN: Bidirectional 3D Human Motion Prediction GAN

Learning Semantic Latent Directions for Accurate and Controllable Human Motion Prediction

Executing Your Commands Via Motion Diffusion in Latent Space

Multi-Resolution Generative Modeling of Human Motion from Limited Data

LEAD: Latent Realignment for Human Motion Diffusion

3D Skeleton-based Human Motion Prediction with Manifold-Aware GAN

Generative Human Motion Stylization in Latent Space

Human Motion Prediction Using Manifold-Aware Wasserstein GAN

Make-An-Animation: Large-Scale Text-conditional 3D Human Motion Generation

Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs

Controllable GAN Synthesis Using Non-Rigid Structure-from-Motion