Abstract:We present a complete system for real-time rendering of scenes with complex appearance previously reserved for offline use. This is achieved with a combination of algorithmic and system level innovations. Our appearance model utilizes learned hierarchical textures that are interpreted using neural decoders, which produce reflectance values and importance-sampled directions. To best utilize the modeling capacity of the decoders, we equip the decoders with two graphics priors. The first prior -- transformation of directions into learned shading frames -- facilitates accurate reconstruction of mesoscale effects. The second prior -- a microfacet sampling distribution -- allows the neural decoder to perform importance sampling efficiently. The resulting appearance model supports anisotropic sampling and level-of-detail rendering, and allows baking deeply layered material graphs into a compact unified neural representation. By exposing hardware accelerated tensor operations to ray tracing shaders, we show that it is possible to inline and execute the neural decoders efficiently inside a real-time path tracer. We analyze scalability with increasing number of neural materials and propose to improve performance using code optimized for coherent and divergent execution. Our neural material shaders can be over an order of magnitude faster than non-neural layered materials. This opens up the door for using film-quality visuals in real-time applications such as games and live previews.

What problem does this paper attempt to address?

The main problem this paper attempts to solve is achieving high-quality representation of complex material appearances in real-time rendering. Specifically, the authors aim to develop a complete system capable of real-time rendering of scenes with complex appearances, which were previously only feasible for offline rendering. To achieve this goal, they combine innovations at both the algorithm and system levels, proposing an appearance model that uses a neural decoder to interpret layered textures, capable of generating reflectance values and importance sampling directions. In this way, their system can integrate these complex objects into real-time renderers in a scalable manner while maintaining high fidelity. The specific issues the paper attempts to address are as follows: 1. **Visual Fidelity**: The paper aims to faithfully reproduce a range of challenging materials, including multilayer materials, low-roughness dielectric coatings, conductors with glints, stains, and anisotropic materials. This goes beyond fitting spatially uniform measured material datasets, with particular attention to materials with high-resolution textures (4K and above) and detailed normal maps. 2. **Level of Detail**: Unfiltered high-resolution materials can become severely aliased when displayed at reduced scales, so the paper seeks to support filtered queries to enable level-of-detail rendering with fewer samples. 3. **Importance Sampling**: In addition to representing the BRDF (Bidirectional Reflectance Distribution Function), effective sampling strategies are needed to allow deployment in Monte Carlo estimators such as path tracing, particularly addressing the traditional challenge of importance sampling for filtered versions of materials. 4. **Performance**: The neural representation is intended for real-time applications, where material evaluation may only occupy a small fraction of the total frame time. Therefore, it needs to be compatible with path tracing, i.e., evaluating materials at multiple random locations across bounces, which excludes large networks and convolution-dependent models. 5. **Practicality**: While the optimization of neural materials is conducted offline, the training time must remain reasonable. Even for high-resolution materials (4K and above), training times of several days are unacceptable. Through efforts in the above areas, the paper proposes an efficient, high-quality neural material representation system that meets the demands of real-time applications.

Real-Time Neural Appearance Models

Real-Time Neural Appearance Models

Real‐Time Neural Materials using Block‐Compressed Features

RNA: Relightable Neural Assets

Advances in Neural Rendering

A Hierarchical Architecture for Neural Materials

Rig-space Neural Rendering

Real‐time Neural Rendering of Dynamic Light Fields

Deep Appearance Models for Face Rendering

AdaNeRF: Adaptive Sampling for Real-time Rendering of Neural Radiance Fields

A Flexible Neural Renderer for Material Visualization

Real-Time Neural Rasterization for Large Scenes

Gaussian Time Machine: A Real-Time Rendering Methodology for Time-Variant Appearances

Baking Neural Radiance Fields for Real-Time View Synthesis

Neural Global Illumination Via Superposed Deformable Feature Fields

State of the Art on Neural Rendering

Learning Neural Duplex Radiance Fields for Real-Time View Synthesis

DONeRF: Towards Real‐Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks

Neural Scene Baking for Permutation Invariant Transparency Rendering with Real-time Global Illumination

Neural Super-Resolution in Real-Time Rendering Using Auxiliary Feature Enhancement