Abstract:Visual error metrics play a fundamental role in the quantification of perceived image similarity. Most recently, use cases for them in real-time applications have emerged, such as content-adaptive shading and shading reuse to increase performance and improve efficiency. A wide range of different metrics has been established, with the most sophisticated being capable of capturing the perceptual characteristics of the human visual system. However, their complexity, computational expense, and reliance on reference images to compare against prevent their generalized use in real-time, restricting such applications to using only the simplest available metrics. In this work, we explore the abilities of convolutional neural networks to predict a variety of visual metrics without requiring either reference or rendered images. Specifically, we train and deploy a neural network to estimate the visual error resulting from reusing shading or using reduced shading rates. The resulting models account for 70%-90% of the variance while achieving up to an order of magnitude faster computation times. Our solution combines image-space information that is readily available in most state-of-the-art deferred shading pipelines with reprojection from previous frames to enable an adequate estimate of visual errors, even in previously unseen regions. We describe a suitable convolutional network architecture and considerations for data preparation for training. We demonstrate the capability of our network to predict complex error metrics at interactive rates in a real-time application that implements content-adaptive shading in a deferred pipeline. Depending on the portion of unseen image regions, our approach can achieve up to $2\times$ performance compared to state-of-the-art methods.

Deep and Fast Approximate Order Independent Transparency

Deep and Fast Approximate Order Independent Transparency

GPU accelerated rendering smooth silhouettes

An approximate image-space approach for real-time rendering of deformable translucent objects

Fast, Realistic Lighting and Material Design Using Nonlinear Cut Approximation

Lightweight Bilateral Convolutional Neural Networks for Interactive Single-Bounce Diffuse Indirect Illumination.

Neural Scene Baking for Permutation Invariant Transparency Rendering with Real-time Global Illumination

Deep Illumination: Approximating Dynamic Global Illumination with Generative Adversarial Network

ShaderTransformer: Predicting Shader Quality Via One-shot Embedding for Fast Simplification

Runtime Shader Simplification Via Instant Search in Reduced Optimization Space.

LucidRaster: GPU Software Rasterizer for Exact Order-Independent Transparency

LIT: Light-field Inference of Transparency for Refractive Object Localization

Efficient Visibility Approximation for Game AI using Neural Omnidirectional Distance Fields

Trustworthy Self-Attention: Enabling the Network to Focus Only on the Most Relevant References

Fast Differentiable Transient Rendering for Non-Line-of-Sight Reconstruction

Real-Time Rendering of Heterogeneous Translucent Objects with Arbitrary Shapes

Training and Predicting Visual Error for Real-Time Applications

NeuPreSS: Compact Neural Precomputed Subsurface Scattering for Distant Lighting of Heterogeneous Translucent Objects

Lightweight Bilateral Convolutional Neural Networks for Real-time Diffuse Global Illumination

Edge-Guided Occlusion Fading Reduction for a Light-Weighted Self-Supervised Monocular Depth Estimation

IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination