Abstract:Generating talking person portraits with arbitrary speech audio is a crucial problem in the field of digital human and metaverse. A modern talking face generation method is expected to achieve the goals of generalized audio-lip synchronization, good video quality, and high system efficiency. Recently, neural radiance field (NeRF) has become a popular rendering technique in this field since it could achieve high-fidelity and 3D-consistent talking face generation with a few-minute-long training video. However, there still exist several challenges for NeRF-based methods: 1) as for the lip synchronization, it is hard to generate a long facial motion sequence of high temporal consistency and audio-lip accuracy; 2) as for the video quality, due to the limited data used to train the renderer, it is vulnerable to out-of-domain input condition and produce bad rendering results occasionally; 3) as for the system efficiency, the slow training and inference speed of the vanilla NeRF severely obstruct its usage in real-world applications. In this paper, we propose GeneFace++ to handle these challenges by 1) utilizing the pitch contour as an auxiliary feature and introducing a temporal loss in the facial motion prediction process; 2) proposing a landmark locally linear embedding method to regulate the outliers in the predicted motion sequence to avoid robustness issues; 3) designing a computationally efficient NeRF-based motion-to-video renderer to achieves fast training and real-time inference. With these settings, GeneFace++ becomes the first NeRF-based method that achieves stable and real-time talking face generation with generalized audio-lip synchronization. Extensive experiments show that our method outperforms state-of-the-art baselines in terms of subjective and objective evaluation. Video samples are available at <a class="link-external link-https" href="https://genefaceplusplus.github.io" rel="external noopener nofollow">this https URL</a> .

PortraitNeRF: A Single Neural Radiance Field for Complete and Coordinated Talking Portrait Generation

ER-NeRF++: Efficient region-aware Neural Radiance Fields for high-fidelity talking portrait synthesis

Expression-aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesis

AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis

Real-time Neural Radiance Talking Portrait Synthesis Via Audio-spatial Decomposition

AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesis

GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis

GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation

NeRF-AD: Neural Radiance Field with Attention-based Disentanglement for Talking Face Synthesis

S^3D-NeRF: Single-Shot Speech-Driven Neural Radiance Field for High Fidelity Talking Head Synthesis

One-Shot High-Fidelity Talking-Head Synthesis with Deformable Neural Radiance Field

DT-NeRF: Decomposed Triplane-Hash Neural Radiance Fields for High-Fidelity Talking Portrait Synthesis

TalkinNeRF: Animatable Neural Fields for Full-Body Talking Humans

Audio-driven Talking Face Video Generation with Natural Head Pose

DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Rendering

Real-time 3D-aware Portrait Video Relighting

Embedded Representation Learning Network for Animating Styled Video Portrait

JEAN: Joint Expression and Audio-guided NeRF-based Talking Face Generation

SD-NeRF: Towards Lifelike Talking Head Animation Via Spatially-Adaptive Dual-Driven NeRFs

Controllable Dynamic Appearance for Neural 3D Portraits

DialogueNeRF: Towards Realistic Avatar Face-to-Face Conversation Video Generation