Abstract:It has been shown that rendering in the gradient domain, i.e., estimating finite difference gradients of image intensity using correlated samples, and combining them with direct estimates of pixel intensities by solving a screened Poisson problem, often offers fundamental benefits over merely sampling pixel intensities. The reasons can be traced to the frequency content of the light transport integrand and its interplay with the gradient operator. However, while they often yield state of the art performance among algorithms that are based on Monte Carlo sampling alone, gradient-domain rendering algorithms have, until now, not generally been competitive with techniques that combine Monte Carlo sampling with post-hoc noise removal using sophisticated non-linear filtering. Drawing on the power of modern convolutional neural networks, we propose a novel reconstruction method for gradient-domain rendering. Our technique replaces the screened Poisson solver of previous gradient-domain techniques with a novel dense variant of the U-Net autoencoder, additionally taking auxiliary feature buffers as inputs. We optimize our network to minimize a perceptual image distance metric calibrated to the human visual system. Our results significantly improve the quality obtained from gradient-domain path tracing, allowing it to overtake state-of-the-art comparison techniques that denoise traditional Monte Carlo samplings. In particular, we observe that the correlated gradient samples --- that offer information about the smoothness of the integrand unavailable in standard Monte Carlo sampling --- notably improve image quality compared to an equally powerful neural model that does not make use of gradient samples.

Adaptive Sampling and Reconstruction for Gradient-Domain Rendering

Deep convolutional reconstruction for gradient-domain rendering

Filtering-Based Reconstruction for Gradient-Domain Rendering

Adaptive Incident Radiance Field Sampling and Reconstruction Using Deep Reinforcement Learning.

Unsupervised Image Reconstruction for Gradient‐Domain Volumetric Rendering

Using Adaptive Gradient for Texture Learning in Single-View 3D Reconstruction

Gradient-Based Interpolation and Sampling for Real-Time Rendering of Inhomogeneous, Single-Scattering Media

High-Quality Real-Time Rendering Using Subpixel Sampling Reconstruction

Adaptive Matrix Column Sampling and Completion for Rendering Participating Media.

Error-aware Sampling in Adaptive Shells for Neural Surface Reconstruction

A Novel Ray-Casting Algorithm Using Dynamic Adaptive Sampling

Gradient Estimation for Real-time Adaptive Temporal Filtering

Depth Super-Resolution Using Joint Adaptive Weighted Least Squares and Patching Gradient.

Lossless Basis Expansion for Gradient‐Domain Rendering

Differentiable Rendering with Reparameterized Volume Sampling

Deep Point Set Resampling via Gradient Fields

Langevin monte carlo rendering with gradient-based adaptation

Amortizing Samples in Physics-Based Inverse Rendering Using ReSTIR

Super-Pixel Sampler: a Data-driven Approach for Depth Sampling and Reconstruction

Adaptive Dictionary Learning in Sparse Gradient Domain for Image Recovery

ProbIBR: Fast Image-Based Rendering with Learned Probability-Guided Sampling