Abstract:The generation of global illumination in real time has been a long-standing challenge in the graphics community, particularly in dynamic scenes with complex illumination. Recent neural rendering techniques have shown great promise by utilizing neural networks to represent the illumination of scenes and then decoding the final radiance. However, incorporating object parameters into the representation may limit their effectiveness in handling fully dynamic scenes. This work presents a neural rendering approach, dubbed LightFormer , that can generate realistic global illumination for fully dynamic scenes, including dynamic lighting, materials, cameras, and animated objects, in real time. Inspired by classic many-lights methods, the proposed approach focuses on the neural representation of light sources in the scene rather than the entire scene, leading to the overall better generalizability. The neural prediction is achieved by leveraging the virtual point lights and shading clues for each light. Specifically, two stages are explored. In the light encoding stage, each light generates a set of virtual point lights in the scene, which are then encoded into an implicit neural light representation, along with screen-space shading clues like visibility. In the light gathering stage, a pixel-light attention mechanism composites all light representations for each shading point. Given the geometry and material representation, in tandem with the composed light representations of all lights, a lightweight neural network predicts the final radiance. Experimental results demonstrate that the proposed LightFormer can yield reasonable and realistic global illumination in fully dynamic scenes with real-time performance.

A Multiresolution Network Architecture for Deferred Neural Lighting

Video Super-Resolution Based on Multiple Networks Merging

Deferred Neural Lighting: Free-viewpoint Relighting from Unstructured Photographs

Neural Global Illumination Via Superposed Deformable Feature Fields

De‐NeRF: Ultra‐high‐definition NeRF with deformable net alignment

Hyb-NeRF: A Multiresolution Hybrid Encoding for Neural Radiance Fields

A Hierarchical Architecture for Neural Materials

Adaptive Multi-NeRF: Exploit Efficient Parallelism in Adaptive Multiple Scale Neural Radiance Field Rendering

Multiscale Representation for Real-Time Anti-Aliasing Neural Rendering

AdaNeRF: Adaptive Sampling for Real-time Rendering of Neural Radiance Fields

Deep HDR Imaging via A Non-local Network

Efficient View Synthesis with Neural Radiance Distribution Field

Learning Neural Duplex Radiance Fields for Real-Time View Synthesis

DeRenderNet: Intrinsic Image Decomposition of Urban Scenes with Shape-(In)dependent Shading Rendering

LightFormer: Light-Oriented Global Neural Rendering in Dynamic Scene

ScaNeRF: Scalable Bundle-Adjusting Neural Radiance Fields for Large-Scale Scene Rendering

DONeRF: Towards Real‐Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks

HDR-Net-Fusion: Real-time 3D dynamic scene reconstruction with a hierarchical deep reinforcement network

Real‐time Neural Rendering of Dynamic Light Fields