Motion Avatar: Generate Human and Animal Avatars with Arbitrary Motion

Zeyu Zhang,Yiran Wang,Biao Wu,Shuo Chen,Zhiyuan Zhang,Shiya Huang,Wenbo Zhang,Meng Fang,Ling Chen,Yang Zhao

2024-08-31

Abstract:In recent years, there has been significant interest in creating 3D avatars and motions, driven by their diverse applications in areas like film-making, video games, AR/VR, and human-robot interaction. However, current efforts primarily concentrate on either generating the 3D avatar mesh alone or producing motion sequences, with integrating these two aspects proving to be a persistent challenge. Additionally, while avatar and motion generation predominantly target humans, extending these techniques to animals remains a significant challenge due to inadequate training data and methods. To bridge these gaps, our paper presents three key contributions. Firstly, we proposed a novel agent-based approach named Motion Avatar, which allows for the automatic generation of high-quality customizable human and animal avatars with motions through text queries. The method significantly advanced the progress in dynamic 3D character generation. Secondly, we introduced a LLM planner that coordinates both motion and avatar generation, which transforms a discriminative planning into a customizable Q&A fashion. Lastly, we presented an animal motion dataset named Zoo-300K, comprising approximately 300,000 text-motion pairs across 65 animal categories and its building pipeline ZooGen, which serves as a valuable resource for the community. See project website <a class="link-external link-https" href="https://steve-zeyu-zhang.github.io/MotionAvatar/" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address several key challenges in the field of dynamic 3D character generation:

1. **Integration of 3D Characters and Actions**: Current technologies often focus on generating 3D character models or action sequences separately, but effectively combining the two remains a significant challenge. The paper proposes a new agent-based approach (Motion Avatar) that can automatically generate high-quality, customizable human and animal characters and their actions through text queries.

2. **Generation of Animal Characters**: Although the technology for generating human characters is relatively mature, extending these techniques to animal characters faces significant challenges, mainly due to the lack of sufficient training data and methodological limitations. The paper introduces the Zoo-300K dataset and the ZooGen construction pipeline, providing valuable resources for the generation of animal characters.

3. **Flexibility and Controllability of Dynamic Character Generation**: Traditional generation methods have shortcomings in precise control and multi-view consistency, which limits the application of dynamic characters in interactive scenarios. The paper's proposed LLM planner can coordinate the generation of characters and actions, making the generation process more flexible and controllable, and suitable for a wider range of dynamic character generation tasks.

In summary, the paper aims to address the integration, extension, and control issues in 3D character generation through technological innovation, making significant progress, especially in the generation of animal characters.