Abstract:For several decades, researchers have been advancing techniques for creating and rendering 3D digital faces, where a lot of the effort has gone into geometry and appearance capture, modeling and rendering techniques. This body of research work has largely focused on facial skin, with much less attention devoted to peripheral components like hair, eyes and the interior of the mouth. As a result, even with the best technology for facial capture and rendering, in most high-end productions a lot of artist time is still spent modeling the missing components and fine-tuning the rendering parameters to combine everything into photo-real digital renders. In this work we propose to combine incomplete, high-quality renderings showing only facial skin with recent methods for neural rendering of faces, in order to automatically and seamlessly create photo-realistic full-head portrait renders from captured data without the need for artist intervention. Our method begins with traditional face rendering, where the skin is rendered with the desired appearance, expression, viewpoint, and illumination. These skin renders are then projected into the latent space of a pre-trained neural network that can generate arbitrary photo-real face images (StyleGAN2). The result is a sequence of realistic face images that match the identity and appearance of the 3D character at the skin level, but is completed naturally with synthesized hair, eyes, inner mouth and surroundings. Notably, we present the first method for multi-frame consistent projection into this latent space, allowing photo-realistic rendering and preservation of the identity of the digital human over an animated performance sequence, which can depict different expressions, lighting conditions and viewpoints. Our method can be used in new face rendering pipelines and, importantly, in other deep learning applications that require large amounts of realistic training data with ground-truth 3D geometry, appearance maps, lighting, and viewpoint.

SUPER: Selfie Undistortion and Head Pose Editing with Identity Preservation

Realistic Face Reenactment Via Self-Supervised Disentangling of Identity and Pose

Robust and Accurate 3D Self-Portraits in Seconds

IDE-3D: Interactive Disentangled Editing for High-Resolution 3D-aware Portrait Synthesis

DisCO: Portrait Distortion Correction with Perspective-Aware 3D GANs

Re-Compositable Panoramic Selfie With Robust Multi-Frame Segmentation And Stitching

Recapture as You Want

AgileGAN3D: Few-Shot 3D Portrait Stylization by Augmented Transfer Learning

AvatarMe++: Facial Shape and BRDF Inference With Photorealistic Rendering-Aware GANs

Rendering with style

Flexible Portrait Image Editing with Fine-Grained Control

FaceStudio: Put Your Face Everywhere in Seconds

3D GAN Inversion with Facial Symmetry Prior

High-Fidelity 3D Digital Human Head Creation from RGB-D Selfies

Selfie Retoucher: Subject-Oriented Self-Portrait Enhancement

3DFaceShop: Explicitly Controllable 3D-Aware Portrait Generation

Coherent 3D Portrait Video Reconstruction via Triplane Fusion

CapHuman: Capture Your Moments in Parallel Universes

HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and Retarget Faces

Enhancing the Authenticity of Rendered Portraits with Identity-Consistent Transfer Learning