Abstract:We present a novel framework for free-viewpoint facial performance relighting using diffusion-based image-to-image translation. Leveraging a subject-specific dataset containing diverse facial expressions captured under various lighting conditions, including flat-lit and one-light-at-a-time (OLAT) scenarios, we train a diffusion model for precise lighting control, enabling high-fidelity relit facial images from flat-lit inputs. Our framework includes spatially-aligned conditioning of flat-lit captures and random noise, along with integrated lighting information for global control, utilizing prior knowledge from the pre-trained Stable Diffusion model. This model is then applied to dynamic facial performances captured in a consistent flat-lit environment and reconstructed for novel-view synthesis using a scalable dynamic 3D Gaussian Splatting method to maintain quality and consistency in the relit results. In addition, we introduce unified lighting control by integrating a novel area lighting representation with directional lighting, allowing for joint adjustments in light size and direction. We also enable high dynamic range imaging (HDRI) composition using multiple directional lights to produce dynamic sequences under complex lighting conditions. Our evaluations demonstrate the models efficiency in achieving precise lighting control and generalizing across various facial expressions while preserving detailed features such as skintexture andhair. The model accurately reproduces complex lighting effects like eye reflections, subsurface scattering, self-shadowing, and translucency, advancing photorealism within our framework.

SPLiT: Single Portrait Lighting Estimation via a Tetrad of Face Intrinsics

NeIF: Representing General Reflectance as Neural Intrinsics Fields for Uncalibrated Photometric Stereo

Faces as Lighting Probes via Unsupervised Deep Highlight Extraction

SPLINE-Net: Sparse Photometric Stereo through Lighting Interpolation and Normal Estimation Networks

Intrinsic Face Image Decomposition With Human Face Priors

Deep Lighting Environment Map Estimation from Spherical Panoramas

DifFRelight: Diffusion-Based Facial Performance Relighting

Depth Estimation Algorithm for Light Field Data by Epipolar Image Analysis and Region Interpolation

Learning Inter- and Intra-frame Representations for Non-Lambertian Photometric Stereo

Light-field-depth-estimation Network Based on Epipolar Geometry and Image Segmentation.

Spatiotemporally Consistent HDR Indoor Lighting Estimation

Normal and Visibility Estimation of Human Face from a Single Image

Physically-Based Editing of Indoor Scene Lighting from a Single Image

From Faces to Outdoor Light Probes

Robust Geometry and Reflectance Disentanglement for 3D Face Reconstruction from Sparse-view Images

Relighting from a Single Image: Datasets and Deep Intrinsic-based Architecture

Joint Learning of Portrait Intrinsic Decomposition and Relighting

Spatially-Varying Outdoor Lighting Estimation from Intrinsics

Single image portrait relighting via explicit multiple reflectance channel modeling

Ear-to-ear Capture of Facial Intrinsics

Semi-Supervised Learning: Structure, Reflectance and Lighting Estimation From a Night Image Pair