LEIFR-Net: light estimation for implicit face relight network

Zhiru Li,Chenchu Rong,Yuanqing Wang

DOI: https://doi.org/10.1364/oe.510060

IF: 3.8

2024-01-30

Optics Express

Abstract:Zhiru Li, Chenchu Rong, Yuanqing Wang Relighting facial images based on estimated lighting distribution and intensity from image backgrounds and environments can lead to more ... [Opt. Express 32, 4827-4838 (2024)]

optics

What problem does this paper attempt to address?

NeIF: Representing General Reflectance as Neural Intrinsics Fields for Uncalibrated Photometric Stereo

Zongrui Li,Qian Zheng,Feishi Wang,Boxin Shi,Gang Pan,Xudong Jiang

DOI: https://doi.org/10.48550/arxiv.2208.08897

2022-01-01

Abstract: Uncalibrated photometric stereo (UPS) is challenging due to the inherent ambiguity brought by unknown light. Existing solutions alleviate the ambiguity by either explicitly associating reflectance to light conditions or resolving light conditions in a supervised manner. This paper establishes an implicit relation between light clues and light estimation and solves UPS in an unsupervised manner. The key idea is to represent the reflectance as four neural intrinsics fields, i.e., position, light, specular, and shadow, based on which the neural light field is implicitly associated with light clues of specular reflectance and cast shadow. The unsupervised, joint optimization of neural intrinsics fields can be free from training data bias as well as accumulating error, and fully exploits all observed pixel values for UPS. Our method achieves a superior performance advantage over state-of-the-art UPS methods on public and self-collected datasets, under regular and challenging setups. The code will be released soon.
Designing An Illumination-Aware Network for Deep Image Relighting

Zuo-Liang Zhu,Zhen Li,Rui-Xun Zhang,Chun-Le Guo,Ming-Ming Cheng

DOI: https://doi.org/10.1109/TIP.2022.3195366

2022-07-22

Abstract:Lighting is a determining factor in photography that affects the style, expression of emotion, and even quality of images. Creating or finding satisfying lighting conditions, in reality, is laborious and time-consuming, so it is of great value to develop a technology to manipulate illumination in an image as post-processing. Although previous works have explored techniques based on the physical viewpoint for relighting images, extensive supervisions and prior knowledge are necessary to generate reasonable images, restricting the generalization ability of these works. In contrast, we take the viewpoint of image-to-image translation and implicitly merge ideas of the conventional physical viewpoint. In this paper, we present an Illumination-Aware Network (IAN) which follows the guidance from hierarchical sampling to progressively relight a scene from a single image with high efficiency. In addition, an Illumination-Aware Residual Block (IARB) is designed to approximate the physical rendering process and to extract precise descriptors of light sources for further manipulations. We also introduce a depth-guided geometry encoder for acquiring valuable geometry- and structure-related representations once the depth information is available. Experimental results show that our proposed method produces better quantitative and qualitative relighting results than previous state-of-the-art methods. The code and models are publicly available on

Computer Science
Relighting Neural Radiance Fields with Shadow and Highlight Hints

Chong Zeng,Guojun Chen,Yue Dong,Pieter Peers,Hongzhi Wu,Xin Tong

DOI: https://doi.org/10.1145/3588432.3591482

2023-08-25

Abstract:This paper presents a novel neural implicit radiance representation for free viewpoint relighting from a small set of unstructured photographs of an object lit by a moving point light source different from the view position. We express the shape as a signed distance function modeled by a multi layer perceptron. In contrast to prior relightable implicit neural representations, we do not disentangle the different reflectance components, but model both the local and global reflectance at each point by a second multi layer perceptron that, in addition, to density features, the current position, the normal (from the signed distace function), view direction, and light position, also takes shadow and highlight hints to aid the network in modeling the corresponding high frequency light transport effects. These hints are provided as a suggestion, and we leave it up to the network to decide how to incorporate these in the final relit result. We demonstrate and validate our neural implicit representation on synthetic and real scenes exhibiting a wide variety of shapes, material properties, and global illumination light transport.

Computer Vision and Pattern Recognition,Graphics
A real-time global relighting algorithm

WU Xiang-Yang,Peng Qun-sheng,Zhang Hong-xin,Chen Wei,Liu Gang

2005-01-01

Ruan Jian Xue Bao/Journal of Software

Abstract:In this paper we propose a real-time global relighting algorithm. The scene illumination is divided into three parts: direct, indirect and ambient components. After calculating the reflectance of objects from reconstructed geometry and pre-sampled images, indirect and ambient components of each sampled image are fitted with low-order basis functions. To relight the scene under new illumination conditions or viewpoints, the recovered reflectance is applied to the illumination model to get direct lighting, while indirect and ambient ones are obtained from the basis functions. In addition surface details are recovered by the variations of the micro-property and orientations on the surface computed from sampled images. Our results show that this algorithm can reconstruct global illumination with clear shadow and surface detail vividly in real-time.
Relit-NeuLF: Efficient Relighting and Novel View Synthesis Via Neural 4D Light Field

Zhong Li,Liangchen Song,Zhang Chen,Xiangyu Du,Lele Chen,Junsong Yuan,Yi Xu

DOI: https://doi.org/10.1145/3581783.3612160

2023-01-01

Abstract:In this paper, we address the problem of simultaneous relighting and novel view synthesis of a complex scene from multi-view images with a limited number of light sources. We propose an analysis-synthesis approach called Relit-NeuLF. Following the recent neural 4D light field network (NeuLF)[22], Relit-NeuLF first leverages a two-plane light field representation to parameterize each ray in a 4D coordinate system, enabling efficient learning and inference. Then, we recover the spatially-varying bidirectional reflectance distribution function (SVBRDF) of a 3D scene in a self-supervised manner. A DecomposeNet learns to map each ray to its SVBRDF components: albedo, normal, and roughness. Based on the decomposed BRDF components and conditioning light directions, a RenderNet learns to synthesize the color of the ray. To self-supervise the SVBRDF decomposition, we encourage the predicted ray color to be close to the physically-based rendering result using the microfacet model. Comprehensive experiments demonstrate that the proposed method is efficient and effective on both synthetic data and real-world human face data, and outperforms the state-of-the-art results.
Estimating Illumination Parameters in Real Space with Application to Image Relighting

Feng Xie,Linmi Tao,Guangyou Xu,Huijun Di

DOI: https://doi.org/10.1007/11612032_50

2006-01-01

Abstract:A novel algorithm is presented in this paper for estimating the direction and strength of point light with the strength of ambient illumination. Existing approaches estimate these illumination parameters directly in the high dimensional image space, while we estimate the parameters in two steps: first to project the image to an orthogonal linear subspace based on spherical harmonic basis functions; then to calculate the parameters in the low dimensional subspace. The experimental results on CMU PIE database and Yale Database B showed the stability and effectiveness of this method. The resulting illumination information can help to synthesize more realistic relighting images and recognize object under variable illumination.
Image-based Relighting from a Sparse Set of Outdoor Images.

Xuehong Zhou,Guanyu Xing,Zhipeng Ding,Yanli Liu,Junjun Xiong,Qunsheng Peng

DOI: https://doi.org/10.1016/j.cag.2013.10.030

IF: 1.821

2013-01-01

Computers & Graphics

Abstract:We propose a novel method for relighting the image of outdoor scenes viewed from a fixed camera based on a sparse set of images of the same scene under different illumination. Unlike previous methods which require capturing images under pre-designed lighting or employing the 3D model of the target objects, our method adopts the technique of basis image which encapsulates material and geometry information of the scene into one image. We present a new method to calculate the basis images of an outdoor scene based on the sampling images captured in different time of a day, the relighting images corresponding to new sunlight incidence directions with arbitrary intensity can then be generated with these basis images. Besides, the subordinate shadow effect adhering to the sun's movement is also simulated, producing a visually plausible relighting result. Experiments demonstrate the efficiency and validity of our approach.
ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering

Rengan Xie,Kai Huang,In-Young Cho,Sen Yang,Wei Chen,Hujun Bao,Wenting Zheng,Rong Li,Yuchi Huo

DOI: https://doi.org/10.1145/3678002

IF: 6.2

2024-01-01

ACM Transactions on Graphics

Abstract:Recently, implicit neural representation has been widely used to learn the appearance of human bodies in the canonical space, which can be further animated using a parametric human model. However, how to decompose the material properties from the implicit representation for relighting has not yet been investigated thoroughly. We propose to address this problem with a novel framework, ReN Human, that takes sparse or even monocular input videos collected in unconstrained lighting to produce a 3D human representation that can be rendered with novel views, poses, and lighting. Our method represents humans as deformable implicit neural representation and decomposes the geometry, material of humans as well as environment illumination for capturing a relightable and animatable human model. Moreover, we introduce a volumetric lighting grid consisting of spherical Gaussian mixtures to learn the spatially varying illumination and animatable visibility probes to model the dynamic self-occlusion caused by human motion. Specifically, we learn the material property fields and illumination using a physically-based rendering layer that uses Monte Carlo importance sampling to facilitate differentiation of the complex rendering integral. We demonstrate that our approach outperforms recent novel views and poses synthesis methods in a challenging benchmark with sparse videos, enabling high-fidelity human relighting.
NeLF: Neural Light-transport Field for Portrait View Synthesis and Relighting

TIANCHENG CHEN

DOI: https://doi.org/10.31219/osf.io/gsfbd

2021-09-11

Abstract:Human portraits exhibit various appearances when observed from different views under different lighting conditions. We can easily imagine how the face will look like in another setup, but computer algorithms still fail on this problem given limited observations. To this end, we present a system for portrait view synthesis and relighting: given multiple portraits, we use a neural network to predict the light-transport field in 3D space, and from the predicted Neural Light-transport Field (NeLF)produce a portrait from a new camera view under a new environmental lighting. Our system is trained on a large number of synthetic models, and can generalize to different synthetic and real portraits under various lighting conditions. Our method achieves simultaneous view synthesis and relighting given multi-view portraits as the input, and achieves state-of-the-art results.
Relighting Abstracted Image Via Salient Edge-Guided Luminance Field Optimization.

Chunxiao Liu,Hong Li,Qunsheng Peng,Xun Wang,Enhua Wu

DOI: https://doi.org/10.1002/cav.1516

IF: 1.01

2013-01-01

Computer Animation and Virtual Worlds

Abstract:Because existing image abstraction systems can hardly incorporate with the changing light, we present an integrated image abstraction and relighting rendering system, which is based on a salient edge-guided luminance field optimization approach. For an input image, we first adopt a sparsity prior-based illumination decomposition method to remove its original illumination and have an intrinsic image. Meanwhile, we iteratively extract the salient edge inside by employing a message-passing strategy. Then, we simplify this image with a proposed salient edge-guided image abstraction optimization algorithm in the luminance field. Finally, we put forward a salient edge-guided image relighting optimization method to simulate the effect of dynamic lighting along different directions. Experiment results show that our system can artistically adjust the illumination of the abstracted image and makes it more vivid. Copyright (c) 2013 John Wiley & Sons, Ltd.
Face Image Relighting Using Locally Constrained Global Optimization

Jiansheng Chen,Guangda Su,Jinping He,Shenglan Ben

DOI: https://doi.org/10.1007/978-3-642-15561-1_4

2010-01-01

Abstract:A face image relighting method using locally constrained global optimization is presented in this paper. Based on the empirical fact that common radiance environments are locally homogeneous, we propose to use an optimization based solution in which local linear adjustments are performed on overlapping windows throughout the input image. As such, local textures and global smoothness of the input image can be preserved simultaneously when applying the illumination transformation. Experimental results demonstrate the effectiveness of the proposed method comparing to some previous approaches.
IllumiNeRF: 3D Relighting without Inverse Rendering

Xiaoming Zhao,Pratul P. Srinivasan,Dor Verbin,Keunhong Park,Ricardo Martin Brualla,Philipp Henzler

2024-06-11

Abstract:Existing methods for relightable view synthesis -- using a set of images of an object under unknown lighting to recover a 3D representation that can be rendered from novel viewpoints under a target illumination -- are based on inverse rendering, and attempt to disentangle the object geometry, materials, and lighting that explain the input images. Furthermore, this typically involves optimization through differentiable Monte Carlo rendering, which is brittle and computationally-expensive. In this work, we propose a simpler approach: we first relight each input image using an image diffusion model conditioned on lighting and then reconstruct a Neural Radiance Field (NeRF) with these relit images, from which we render novel views under the target lighting. We demonstrate that this strategy is surprisingly competitive and achieves state-of-the-art results on multiple relighting benchmarks. Please see our project page at <a class="link-external link-https" href="https://illuminerf.github.io/" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Artificial Intelligence,Graphics
DifFRelight: Diffusion-Based Facial Performance Relighting

Mingming He,Pascal Clausen,Ahmet Levent Taşel,Li Ma,Oliver Pilarski,Wenqi Xian,Laszlo Rikker,Xueming Yu,Ryan Burgert,Ning Yu,Paul Debevec

DOI: https://doi.org/10.1145/3680528.3687644

2024-10-11

Abstract:We present a novel framework for free-viewpoint facial performance relighting using diffusion-based image-to-image translation. Leveraging a subject-specific dataset containing diverse facial expressions captured under various lighting conditions, including flat-lit and one-light-at-a-time (OLAT) scenarios, we train a diffusion model for precise lighting control, enabling high-fidelity relit facial images from flat-lit inputs. Our framework includes spatially-aligned conditioning of flat-lit captures and random noise, along with integrated lighting information for global control, utilizing prior knowledge from the pre-trained Stable Diffusion model. This model is then applied to dynamic facial performances captured in a consistent flat-lit environment and reconstructed for novel-view synthesis using a scalable dynamic 3D Gaussian Splatting method to maintain quality and consistency in the relit results. In addition, we introduce unified lighting control by integrating a novel area lighting representation with directional lighting, allowing for joint adjustments in light size and direction. We also enable high dynamic range imaging (HDRI) composition using multiple directional lights to produce dynamic sequences under complex lighting conditions. Our evaluations demonstrate the models efficiency in achieving precise lighting control and generalizing across various facial expressions while preserving detailed features such as skintexture andhair. The model accurately reproduces complex lighting effects like eye reflections, subsurface scattering, self-shadowing, and translucency, advancing photorealism within our framework.

Computer Vision and Pattern Recognition,Artificial Intelligence,Graphics
Towards High Fidelity Face Relighting with Realistic Shadows

Andrew Hou,Ze Zhang,Michel Sarkis,Ning Bi,Yiying Tong,Xiaoming Liu

DOI: https://doi.org/10.48550/arXiv.2104.00825

2021-06-06

Abstract:Existing face relighting methods often struggle with two problems: maintaining the local facial details of the subject and accurately removing and synthesizing shadows in the relit image, especially hard shadows. We propose a novel deep face relighting method that addresses both problems. Our method learns to predict the ratio (quotient) image between a source image and the target image with the desired lighting, allowing us to relight the image while maintaining the local facial details. During training, our model also learns to accurately modify shadows by using estimated shadow masks to emphasize on the high-contrast shadow borders. Furthermore, we introduce a method to use the shadow mask to estimate the ambient light intensity in an image, and are thus able to leverage multiple datasets during training with different global lighting intensities. With quantitative and qualitative evaluations on the Multi-PIE and FFHQ datasets, we demonstrate that our proposed method faithfully maintains the local facial details of the subject and can accurately handle hard shadows while achieving state-of-the-art face relighting performance.

Computer Vision and Pattern Recognition
Single-image reflectance estimation for relighting by iterative soft grouping

Yuanzhen Li,Stephen Lin,Sing Bing Kang,Hanqing Lu,Heung-Yeung Shum

DOI: https://doi.org/10.1109/PCCGA.2002.1167909

2002-01-01

Abstract:Reflectance values for image-based relighting are often estimated from grouped pixels with similar reflectance, but such groupings are difficult to compute with certainty for sparse image data. To address this problem, we propose an iterative method that aggregates BRDF data in a single image with known geometry and lighting by soft grouping, where pixels contribute to one another's estimate according to their degree of reflectance similarity. Estimation of specular reflectance is further improved by albedo-independent soft grouping of pixels based on shape continuity. With recovered reflectances, we demonstrate realistic relighting for synthetic and real scenes, including surfaces with spatially-varying reflectance.
NeuLighting: Neural Lighting for Free Viewpoint Outdoor Scene Relighting with Unconstrained Photo Collections.

Quewei Li,Jie Guo,Yang Fei,Feichao Li,Yanwen Guo

DOI: https://doi.org/10.1145/3550469.3555384

2022-01-01

Abstract:We propose NeuLighting, a new framework for free viewpoint outdoor scene relighting from a sparse set of unconstrained in-the-wild photo collections. Our framework represents all the scene components as continuous functions parameterized by MLPs that take a 3D location and the lighting condition as input and output reflectance and necessary outdoor illumination properties. Unlike object-level relighting methods which often leverage training images with controllable and consistent indoor illumination, we concentrate on the more challenging outdoor situation where all the images are captured under arbitrary unknown illumination. The key to our method includes a neural lighting representation that compresses the per-image illumination into a disentangled latent vector, and a new free viewpoint relighting scheme that is robust to arbitrary lighting variations across images. The lighting representation is compressive to explain a wide range of illumination and can be easily fed into the query-based NeuLighting framework, enabling efficient shading effect evaluation under any kind of novel illumination. Furthermore, to produce high-quality cast shadows, we estimate the sun visibility map to indicate the shadow regions according to the scene geometry and the sun direction. Thanks to the flexible and explainable neural lighting representation, our system supports outdoor relighting with many different illumination sources, including natural images, environment maps, and time-lapse videos. The high-fidelity renderings under novel views and illumination prove the superiority of our method against state-of-the-art relighting solutions.
A Pre-convolved Representation for Plug-and-Play Neural Illumination Fields

Yiyu Zhuang,Qi Zhang,Xuan Wang,Hao Zhu,Ying Feng,Xiaoyu Li,Ying Shan,Xun Cao

DOI: https://doi.org/10.1609/aaai.v38i7.28618

2024-01-01

Abstract:Recent advances in implicit neural representation have demonstrated the ability to recover detailed geometry and material from multi-view images. However, the use of simplified lighting models such as environment maps to represent non-distant illumination, or using a network to fit indirect light modeling without a solid basis, can lead to an undesirable decomposition between lighting and material. To address this, we propose a fully differentiable framework named Neural Illumination Fields (NeIF) that uses radiance fields as a lighting model to handle complex lighting in a physically based way. Together with integral lobe encoding for roughness-adaptive specular lobe and leveraging the pre-convolved background for accurate decomposition, the proposed method represents a significant step towards integrating physically based rendering into the NeRF representation. The experiments demonstrate the superior performance of novel-view rendering compared to previous works, and the capability to re-render objects under arbitrary NeRF-style environments opens up exciting possibilities for bridging the gap between virtual and real-world scenes.
Single image portrait relighting via explicit multiple reflectance channel modeling

Zhibo Wang,Xin Yu,Ming Lu,Quan Wang,Chen Qian,Feng Xu

DOI: https://doi.org/10.1145/3414685.3417824

IF: 6.2

2020-12-31

ACM Transactions on Graphics

Abstract:Portrait relighting aims to render a face image under different lighting conditions. Existing methods do not explicitly consider some challenging lighting effects such as specular and shadow, and thus may fail in handling extreme lighting conditions. In this paper, we propose a novel framework that explicitly models multiple reflectance channels for single image portrait relighting, including the facial albedo, geometry as well as two lighting effects, i.e. , specular and shadow. These channels are finally composed to generate the relit results via deep neural networks. Current datasets do not support learning such multiple reflectance channel modeling. Therefore, we present a large-scale dataset with the ground-truths of the channels, enabling us to train the deep neural networks in a supervised manner. Furthermore, we develop a novel module named Lighting guided Feature Modulation (LFM). In contrast to existing methods which simply incorporate the given lighting in the bottleneck of a network, LFM fuses the lighting by layer-wise feature modulation to deliver more convincing results. Extensive experiments demonstrate that our proposed method achieves better results and is able to generate challenging lighting effects.

computer science, software engineering
Object-Based Illumination Estimation with Rendering-Aware Neural Networks

Xin Wei,Guojun Chen,Yue Dong,Stephen Lin,Xin Tong

DOI: https://doi.org/10.1007/978-3-030-58555-6_23

2020-01-01

Abstract:We present a scheme for fast environment light estimation from the RGBD appearance of individual objects and their local image areas. Conventional inverse rendering is too computationally demanding for real-time applications, and the performance of purely learning-based techniques may be limited by the meager input data available from individual objects. To address these issues, we propose an approach that takes advantage of physical principles from inverse rendering to constrain the solution, while also utilizing neural networks to expedite the more computationally expensive portions of its processing, to increase robustness to noisy input data as well as to improve temporal and spatial stability. This results in a rendering-aware system that estimates the local illumination distribution at an object with high accuracy and in real time. With the estimated lighting, virtual objects can be rendered in AR scenarios with shading that is consistent to the real scene, leading to improved realism.
NeAI: A Pre-convoluted Representation for Plug-and-Play Neural Ambient Illumination

Yiyu Zhuang,Qi Zhang,Xuan Wang,Hao Zhu,Ying Feng,Xiaoyu Li,Ying Shan,Xun Cao

2023-01-01

Abstract: Recent advances in implicit neural representation have demonstrated the ability to recover detailed geometry and material from multi-view images. However, the use of simplified lighting models such as environment maps to represent non-distant illumination, or using a network to fit indirect light modeling without a solid basis, can lead to an undesirable decomposition between lighting and material. To address this, we propose a fully differentiable framework named neural ambient illumination (NeAI) that uses Neural Radiance Fields (NeRF) as a lighting model to handle complex lighting in a physically based way. Together with integral lobe encoding for roughness-adaptive specular lobe and leveraging the pre-convoluted background for accurate decomposition, the proposed method represents a significant step towards integrating physically based rendering into the NeRF representation. The experiments demonstrate the superior performance of novel-view rendering compared to previous works, and the capability to re-render objects under arbitrary NeRF-style environments opens up exciting possibilities for bridging the gap between virtual and real-world scenes. The project and supplementary materials are available at https://yiyuzhuang.github.io/NeAI/.

LEIFR-Net: light estimation for implicit face relight network

NeIF: Representing General Reflectance as Neural Intrinsics Fields for Uncalibrated Photometric Stereo

Designing An Illumination-Aware Network for Deep Image Relighting

Relighting Neural Radiance Fields with Shadow and Highlight Hints

A real-time global relighting algorithm

Relit-NeuLF: Efficient Relighting and Novel View Synthesis Via Neural 4D Light Field

Estimating Illumination Parameters in Real Space with Application to Image Relighting

Image-based Relighting from a Sparse Set of Outdoor Images.

ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering

NeLF: Neural Light-transport Field for Portrait View Synthesis and Relighting

Relighting Abstracted Image Via Salient Edge-Guided Luminance Field Optimization.

Face Image Relighting Using Locally Constrained Global Optimization

IllumiNeRF: 3D Relighting without Inverse Rendering

DifFRelight: Diffusion-Based Facial Performance Relighting

Towards High Fidelity Face Relighting with Realistic Shadows

Single-image reflectance estimation for relighting by iterative soft grouping

NeuLighting: Neural Lighting for Free Viewpoint Outdoor Scene Relighting with Unconstrained Photo Collections.

A Pre-convolved Representation for Plug-and-Play Neural Illumination Fields

Single image portrait relighting via explicit multiple reflectance channel modeling

Object-Based Illumination Estimation with Rendering-Aware Neural Networks

NeAI: A Pre-convoluted Representation for Plug-and-Play Neural Ambient Illumination