Abstract:The generation of global illumination in real time has been a long-standing challenge in the graphics community, particularly in dynamic scenes with complex illumination. Recent neural rendering techniques have shown great promise by utilizing neural networks to represent the illumination of scenes and then decoding the final radiance. However, incorporating object parameters into the representation may limit their effectiveness in handling fully dynamic scenes. This work presents a neural rendering approach, dubbed LightFormer , that can generate realistic global illumination for fully dynamic scenes, including dynamic lighting, materials, cameras, and animated objects, in real time. Inspired by classic many-lights methods, the proposed approach focuses on the neural representation of light sources in the scene rather than the entire scene, leading to the overall better generalizability. The neural prediction is achieved by leveraging the virtual point lights and shading clues for each light. Specifically, two stages are explored. In the light encoding stage, each light generates a set of virtual point lights in the scene, which are then encoded into an implicit neural light representation, along with screen-space shading clues like visibility. In the light gathering stage, a pixel-light attention mechanism composites all light representations for each shading point. Given the geometry and material representation, in tandem with the composed light representations of all lights, a lightweight neural network predicts the final radiance. Experimental results demonstrate that the proposed LightFormer can yield reasonable and realistic global illumination in fully dynamic scenes with real-time performance.

Neural LightRig: Unlocking Accurate Object Normal and Material Estimation with Multi-Light Diffusion

Joint Optimization of Triangle Mesh, Material, and Light from Neural Fields with Neural Radiance Cache

VDN-NeRF: Resolving Shape-Radiance Ambiguity Via View-Dependence Normalization

ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering

Neural Gaffer: Relighting Any Object via Diffusion

NeRFactor

NeILF: Neural Incident Light Field for Physically-based Material Estimation

Object-Based Illumination Estimation with Rendering-Aware Neural Networks

IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination

Neural Relighting with Subsurface Scattering by Learning the Radiance Transfer Gradient

NeAI: A Pre-convoluted Representation for Plug-and-Play Neural Ambient Illumination

Neural Global Illumination Via Superposed Deformable Feature Fields

Enhancing Neural Radiance Fields with Depth and Normal Completion Priors from Sparse Views

MLI-NeRF: Multi-Light Intrinsic-Aware Neural Radiance Fields

3D Scene Creation and Rendering via Rough Meshes: A Lighting Transfer Avenue

Aleth-NeRF: Illumination Adaptive NeRF with Concealing Field Assumption

Relit-NeuLF: Efficient Relighting and Novel View Synthesis Via Neural 4D Light Field

Bright-NeRF:Brightening Neural Radiance Field with Color Restoration from Low-light Raw Images

LiDeNeRF: Neural radiance field reconstruction with depth prior provided by LiDAR point cloud

LightFormer: Light-Oriented Global Neural Rendering in Dynamic Scene

NeuralTO: Neural Reconstruction and View Synthesis of Translucent Objects