Abstract:Achieving photorealistic 3D view synthesis and relighting of human portraits is pivotal for advancing AR/VR applications. Existing methodologies in portrait relighting demonstrate substantial limitations in terms of generalization and 3D consistency, coupled with inaccuracies in physically realistic lighting and identity preservation. Furthermore, personalization from a single view is difficult to achieve and often requires multiview images during the testing phase or involves slow optimization processes. This paper introduces Lite2Relight, a novel technique that can predict 3D consistent head poses of portraits while performing physically plausible light editing at interactive speed. Our method uniquely extends the generative capabilities and efficient volumetric representation of EG3D, leveraging a lightstage dataset to implicitly disentangle face reflectance and perform relighting under target HDRI environment maps. By utilizing a pre-trained geometry-aware encoder and a feature alignment module, we map input images into a relightable 3D space, enhancing them with a strong face geometry and reflectance prior. Through extensive quantitative and qualitative evaluations, we show that our method outperforms the state-of-the-art methods in terms of efficacy, photorealism, and practical application. This includes producing 3D-consistent results of the full head, including hair, eyes, and expressions. Lite2Relight paves the way for large-scale adoption of photorealistic portrait editing in various domains, offering a robust, interactive solution to a previously constrained problem. Project page: <a class="link-external link-https" href="https://vcai.mpi-inf.mpg.de/projects/Lite2Relight/" rel="external noopener nofollow">this https URL</a>

Capturing Relightable Human Performances under General Uncontrolled Illumination.

Free-viewpoint video relighting from multi-view sequence under general illumination

ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering

Relighting4D: Neural Relightable Human from Videos

Relightable Neural Actor with Intrinsic Decomposition and Pose Control

DifFRelight: Diffusion-Based Facial Performance Relighting

Seeing people in different light--joint shape, motion, and reflectance capture

Relightable Detailed Human Reconstruction from Sparse Flashlight Images

Computer Graphics in China: A practical approach for real-time illumination estimation of outdoor videos

Online Illumination Estimation of Outdoor Scenes Based on Videos Containing No Shadow Area.

Personalized Video Relighting With an At-Home Light Stage

SAILOR: Synergizing Radiance and Occupancy Fields for Live Human Performance Capture

Relightable and Animatable Neural Avatar from Sparse-View Video

Lite2Relight: 3D-aware Single Image Portrait Relighting

Model-based Outdoor Performance Capture

Free-viewpoint Indoor Neural Relighting from Multi-view Stereo

HDhuman: High-quality Human Performance Capture with Sparse Views

LiveCap: Real-time Human Performance Capture from Monocular Video

Human Performance Capture Using Multiple Handheld Kinects

RelightableHands: Efficient Neural Relighting of Articulated Hand Models

Free-viewpoint video of human actors using multiple handheld Kinects.