Abstract:In this paper, we present a new method to modify the appearance of a face image by manipulating the illumination condition, when the face geometry and albedo information is unknown. This problem is particularly difficult when there is only a single image of the subject available. Recent research demonstrates that the set of images of a convex Lambertian object obtained under a wide variety of lighting conditions can be approximated accurately by a low-dimensional linear subspace using a spherical harmonic representation. Moreover, morphable models are statistical ensembles of facial properties such as shape and texture. In this paper, we integrate spherical harmonics into the morphable model framework by proposing a 3D spherical harmonic basis morphable model (SHBMM). The proposed method can represent a face under arbitrary unknown lighting and pose simply by three low-dimensional vectors, i.e., shape parameters, spherical harmonic basis parameters, and illumination coefficients, which are called the SHBMM parameters. However, when the image was taken under an extreme lighting condition, the approximation error can be large, thus making it difficult to recover albedo information. In order to address this problem, we propose a subregion-based framework that uses a Markov random field to model the statistical distribution and spatial coherence of face texture, which makes our approach not only robust to extreme lighting conditions, but also insensitive to partial occlusions. The performance of our framework is demonstrated through various experimental results, including the improved rates for face recognition under extreme lighting conditions.

A Self-Occlusion Aware Lighting Model for Real-Time Dynamic Reconstruction.

Efficient screen space approach for high complex dynamic scenes in real-time rendering

Dynamic Anisotropic Occlusion

Non-Line-of-Sight Reconstruction using Efficient Transient Rendering

Real-Time Geometry, Albedo, and Motion Reconstruction Using a Single RGB-D Camera

SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video

A real-time global relighting algorithm

Online Illumination Estimation of Outdoor Scenes Based on Videos Containing No Shadow Area.

Robust and Real-Time Outdoor Illumination Estimation

Relightable Detailed Human Reconstruction from Sparse Flashlight Images

Face Relighting from a Single Image under Arbitrary Unknown Lighting Conditions

Multi-view Reconstruction under Varying Illumination Conditions

An Efficient Volumetric Mesh Representation for Real-time Scene Reconstruction using Spatial Hashing

Learning Visibility Field for Detailed 3D Human Reconstruction and Relighting

Occlusion-Free Scene Recovery Via Neural Radiance Fields

Real-Time Globally Consistent 3D Reconstruction with Semantic Priors

Deep Inverse Rendering for Practical Object Appearance Scan with Uncalibrated Illumination

Error Model and Concise Temporal Network for Indirect Illumination in 3D Reconstruction

Real-time Acquisition and Reconstruction of Dynamic Volumes with Neural Structured Illumination

Robust Geometry and Reflectance Disentanglement for 3D Face Reconstruction from Sparse-view Images

Vis2Mesh: Efficient Mesh Reconstruction from Unstructured Point Clouds of Large Scenes with Learned Virtual View Visibility