Abstract:In this paper, we introduce the Volumetric Relightable Morphable Model (VRMM), a novel volumetric and parametric facial prior for 3D face modeling. While recent volumetric prior models offer improvements over traditional methods like 3D Morphable Models (3DMMs), they face challenges in model learning and personalized reconstructions. Our VRMM overcomes these by employing a novel training framework that efficiently disentangles and encodes latent spaces of identity, expression, and lighting into low-dimensional representations. This framework, designed with self-supervised learning, significantly reduces the constraints for training data, making it more feasible in practice. The learned VRMM offers relighting capabilities and encompasses a comprehensive range of expressions. We demonstrate the versatility and effectiveness of VRMM through various applications like avatar generation, facial reconstruction, and animation. Additionally, we address the common issue of overfitting in generative volumetric models with a novel prior-preserving personalization framework based on VRMM. Such an approach enables accurate 3D face reconstruction from even a single portrait input. Our experiments showcase the potential of VRMM to significantly enhance the field of 3D face modeling.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the poor performance of existing 3D facial modeling methods under dynamic expressions and changing lighting conditions. Specifically, although traditional 3D Morphable Models (3DMMs) can encode facial identity, expressions and other attributes, they have difficulties in achieving truly realistic details, especially when dealing with complex head components such as hair and the inside of the mouth. And although the existing volume - based models are more comprehensive in representing facial structures, they still have significant shortcomings in dynamic expression modeling and simulating the impact of different lighting conditions on the face. In addition, these models are prone to over - fitting problems in personalized reconstruction tasks, making it difficult to perform high - quality 3D facial reconstruction from a small number of input images. To this end, the paper proposes a brand - new Volume Relightable Morphable Model (VRMM), aiming to overcome the above challenges. VRMM efficiently decouples and encodes the low - dimensional representation spaces of identity, expression and lighting through the adoption of a self - supervised learning framework, thereby significantly reducing the constraints of training data and making it more practical in practice. In addition, the paper also proposes a VRMM - based prior - preserving personalization framework to solve the over - fitting problem in the generated volume model, enabling accurate 3D facial reconstruction even from a single portrait input. In summary, the main contributions of the paper are as follows: 1. Propose VRMM, which, as far as the authors know, is the first 3D volume facial prior model that can be continuously relighted and contains a complete range of expressions. 2. Design a new training framework for learning the decoupled parameter spaces of expressions, identity and lighting from multi - view image sequences captured under controllable lighting conditions. 3. Propose a novel personalization method, carefully designed to maintain the animatable and relightable characteristics provided by the prior, so as to be able to achieve high - fidelity avatar reconstruction from several or even a single image. 4. Extensive experiments show that VRMM performs excellently in various applications and outperforms previous methods.

VRMM: A Volumetric Relightable Morphable Head Model

On Learning 3D Face Morphable Model from In-the-wild Images

GPHM: Gaussian Parametric Head Model for Monocular Head Avatar Reconstruction

Neural Point-based Volumetric Avatar: Surface-guided Neural Points for Efficient and Photorealistic Volumetric Head Avatar

3D3M: 3D Modulated Morphable Model for Monocular Face Reconstruction

3D Gaussian Parametric Head Model

3DMM-RF: Convolutional Radiance Fields for 3D Face Modeling

Learning Personalized High Quality Volumetric Head Avatars from Monocular RGB Videos

Mm3dface: Nonintrusive 3D Facial Reconstruction Leveraging Mmwave Signals

Multi-view 3D Morphable Face Reconstruction via Canonical Volume Fusion

AvatarMAV: Fast 3D Head Avatar Reconstruction Using Motion-Aware Neural Voxels

MA-NeRF: Motion-Assisted Neural Radiance Fields for Face Synthesis from Sparse Images

MMFace: A Multi-Metric Regression Network for Unconstrained Face Reconstruction

Towards Native Generative Model for 3D Head Avatar

A Modeling Method for the Human Body Model with Facial Morphology

RenderMe-360: A Large Digital Asset Library and Benchmarks Towards High-fidelity Head Avatars

ASM: Adaptive Skinning Model for High-Quality 3D Face Modeling

A New Algorithm for 3d Facial Model Reconstruction and Its Application in Vr

High-fidelity Facial Avatar Reconstruction from Monocular Video with Generative Priors

HAvatar: High-fidelity Head Avatar via Facial Model Conditioned Neural Radiance Field

Semantically Disentangled Variational Autoencoder for Modeling 3D Facial Details