Abstract:We introduce LumiNet, a novel architecture that leverages generative models and latent intrinsic representations for effective lighting transfer. Given a source image and a target lighting image, LumiNet synthesizes a relit version of the source scene that captures the target's lighting. Our approach makes two key contributions: a data curation strategy from the StyleGAN-based relighting model for our training, and a modified diffusion-based ControlNet that processes both latent intrinsic properties from the source image and latent extrinsic properties from the target image. We further improve lighting transfer through a learned adaptor (MLP) that injects the target's latent extrinsic properties via cross-attention and fine-tuning. Unlike traditional ControlNet, which generates images with conditional maps from a single scene, LumiNet processes latent representations from two different images - preserving geometry and albedo from the source while transferring lighting characteristics from the target. Experiments demonstrate that our method successfully transfers complex lighting phenomena including specular highlights and indirect illumination across scenes with varying spatial layouts and materials, outperforming existing approaches on challenging indoor scenes using only images as input.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the **problem of illumination condition transfer between indoor scenes**. Specifically, the authors propose a new architecture named **LUMINET** to synthesize a relit version of the source scene given the source image and the target illumination image, in order to capture the illumination effect of the target image. This problem is of great significance in practical applications, such as in the fields of film production, architectural visualization, and mixed reality. #### Main challenges: 1. **Complex lighting phenomena**: Indoor scenes contain complex light - transfer phenomena, such as specular highlights, shadows, and indirect lighting, which are closely related to the geometric structure and material properties of the scene. 2. **Illumination transfer between different scenes**: The spatial layout and surface properties of different scenes vary greatly, increasing the difficulty of illumination transfer. 3. **Light source identification**: Lighting cannot appear randomly and must come from a reasonable light source position (such as a lamp), so the model needs to understand the light source distribution in the scene. #### Solutions: LUMINET combines generative models and latent intrinsic representations and solves the above challenges in the following ways: 1. **Data preparation**: - Use the variational StyleGAN method to generate diverse illumination - change samples to ensure the diversity of training data. - Utilize real - world datasets (such as Multi - Illumination Images in the Wild (MIIW) and BigTime) to enhance the authenticity and complexity of the training data. 2. **Latent intrinsic extraction**: - Extract latent intrinsic features and illumination codes from images through a pre - trained latent intrinsic encoder, avoiding complex decomposition directly in the pixel space. 3. **Latent intrinsic control**: - Achieve illumination control in the latent space, process latent features through Latent Intrinsic ControlNet, and enhance the illumination control effect through the cross - attention mechanism. 4. **Training objective**: - Conduct illumination transfer training within the same scene through the latent diffusion process to optimize the model's illumination prediction ability. #### Experimental verification: - **Quantitative evaluation**: Extensive quantitative evaluation was carried out on multi - illumination datasets, and the results show that LUMINET significantly outperforms existing methods in all metrics. - **Qualitative evaluation**: Demonstrate the advantages of LUMINET in handling complex lighting phenomena (such as specular highlights, shadows, and indirect lighting) through visual effects. - **User study**: Further verify the effectiveness of LUMINET through user studies. In conclusion, LUMINET provides an efficient and high - quality solution for indoor scene illumination transfer, which can achieve complex illumination effect transfer while maintaining the geometric structure and material properties of the scene.

LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene Relighting

LightIt: Illumination Modeling and Control for Diffusion Models

DiLightNet: Fine-grained Lighting Control for Diffusion-based Image Generation

LumiSculpt: A Consistency Lighting Control Network for Video Generation

Neural Gaffer: Relighting Any Object via Diffusion

Relighting from a Single Image: Datasets and Deep Intrinsic-based Architecture

LumiGAN: Unconditional Generation of Relightable 3D Human Faces

LightenDiffusion: Unsupervised Low-Light Image Enhancement with Latent-Retinex Diffusion Models

Latent Intrinsics Emerge from Training to Relight

Designing An Illumination-Aware Network for Deep Image Relighting

Retinex-Diffusion: On Controlling Illumination Conditions in Diffusion Models via Retinex Theory

ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering

Lightweight Bilateral Convolutional Neural Networks for Interactive Single-Bounce Diffuse Indirect Illumination.

Deep Relighting Networks for Image Light Source Manipulation

Lightweight Bilateral Convolutional Neural Networks for Real-time Diffuse Global Illumination

DifFRelight: Diffusion-Based Facial Performance Relighting

Learning Relighting and Intrinsic Decomposition in Neural Radiance Fields

Relighting Neural Radiance Fields with Shadow and Highlight Hints

DiffMat: Latent diffusion models for image-guided material generation

LightFormer: Light-Oriented Global Neural Rendering in Dynamic Scene

SpotLight: Shadow-Guided Object Relighting via Diffusion