RodinHD: High-Fidelity 3D Avatar Generation with Diffusion Models

Bowen Zhang,Yiji Cheng,Chunyu Wang,Ting Zhang,Jiaolong Yang,Yansong Tang,Feng Zhao,Dong Chen,Baining Guo

2024-07-11

Abstract:We present RodinHD, which can generate high-fidelity 3D avatars from a portrait image. Existing methods fail to capture intricate details such as hairstyles which we tackle in this paper. We first identify an overlooked problem of catastrophic forgetting that arises when fitting triplanes sequentially on many avatars, caused by the MLP decoder sharing scheme. To overcome this issue, we raise a novel data scheduling strategy and a weight consolidation regularization term, which improves the decoder's capability of rendering sharper details. Additionally, we optimize the guiding effect of the portrait image by computing a finer-grained hierarchical representation that captures rich 2D texture cues, and injecting them to the 3D diffusion model at multiple layers via cross-attention. When trained on 46K avatars with a noise schedule optimized for triplanes, the resulting model can generate 3D avatars with notably better details than previous methods and can generalize to in-the-wild portrait input.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The main goal of this paper is to address the problem of high-fidelity 3D avatar generation, particularly the challenges faced when generating detailed 3D avatars from in-the-wild portrait images.

Specifically, the research team proposed the RodinHD method, which aims to generate high-quality, detail-rich 3D avatars from a single portrait image. Existing methods fall short in capturing complex details (such as hairstyles), which limits the practical application value of the generated 3D avatars. To overcome this issue, the RodinHD method focuses on addressing two key challenges:

1. **Catastrophic Forgetting Problem**: When fitting many different 3D avatars consecutively, catastrophic forgetting occurs, causing the decoder to fail to retain previously learned knowledge well, thereby affecting its ability to generate new 3D avatars.

2. **Detail Generation Problem**: Existing 3D generative diffusion models struggle to generate fine details, such as sharp clothing textures and hair strands.

To address the above issues, RodinHD adopts the following strategies:

- **Data Scheduling Strategy and Weight Consolidation Regularization**: Introduces a new data scheduling strategy, namely task replay, and a weight consolidation regularization term to enhance the decoder's ability to render sharp details.

- **Improved Conditional Generation**: Optimizes the effect of guiding portrait images by computing a finer hierarchical representation to capture rich 2D texture cues and injecting these cues into multiple levels of the 3D diffusion model.

- **Noise Schedule Optimization**: Considering the special nature of 3D avatars, optimizes the noise schedule to ensure the model can effectively learn high-resolution 3D avatars.

Through these techniques, RodinHD can generate high-quality 3D avatars with clear clothing textures and hairstyles without additional refinement models. Moreover, this method also demonstrates good cross-view consistency, which is lacking in many other methods.