Abstract:It has been shown that rendering in the gradient domain, i.e., estimating finite difference gradients of image intensity using correlated samples, and combining them with direct estimates of pixel intensities by solving a screened Poisson problem, often offers fundamental benefits over merely sampling pixel intensities. The reasons can be traced to the frequency content of the light transport integrand and its interplay with the gradient operator. However, while they often yield state of the art performance among algorithms that are based on Monte Carlo sampling alone, gradient-domain rendering algorithms have, until now, not generally been competitive with techniques that combine Monte Carlo sampling with post-hoc noise removal using sophisticated non-linear filtering. Drawing on the power of modern convolutional neural networks, we propose a novel reconstruction method for gradient-domain rendering. Our technique replaces the screened Poisson solver of previous gradient-domain techniques with a novel dense variant of the U-Net autoencoder, additionally taking auxiliary feature buffers as inputs. We optimize our network to minimize a perceptual image distance metric calibrated to the human visual system. Our results significantly improve the quality obtained from gradient-domain path tracing, allowing it to overtake state-of-the-art comparison techniques that denoise traditional Monte Carlo samplings. In particular, we observe that the correlated gradient samples --- that offer information about the smoothness of the integrand unavailable in standard Monte Carlo sampling --- notably improve image quality compared to an equally powerful neural model that does not make use of gradient samples.

Transforming a Non-Differentiable Rasterizer into a Differentiable One with Stochastic Gradient Estimation

Rasterized Edge Gradients: Handling Discontinuities Differentiably

Learning to Rasterize Differentiably

Adaptive Sampling and Reconstruction for Gradient-Domain Rendering

Differentiable Rendering Using RGBXY Derivatives and Optimal Transport.

Differentiable Monte Carlo ray tracing through edge sampling

DiffCSG: Differentiable CSG via Rasterization

Generalizing Stochastic Smoothing for Differentiation and Gradient Estimation

Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction

Differentiable Rendering of Neural SDFs through Reparameterization

Stylized Rendering as a Function of Expectation

Deep convolutional reconstruction for gradient-domain rendering

Recursive Control Variates for Inverse Rendering

Differentiable Rendering for Pose Estimation in Proximity Operations

Analytical Derivatives for Differentiable Renderer: 3D Pose Estimation by Silhouette Consistency

RISP: Rendering-Invariant State Predictor with Differentiable Simulation and Rendering for Cross-Domain Parameter Estimation

ZeroGrads: Learning Local Surrogates for Non-Differentiable Graphics

Differentiable Rendering with Reparameterized Volume Sampling

Efficient Differentiation of Pixel Reconstruction Filters for Path-Space Differentiable Rendering

Demystifying SGD with Doubly Stochastic Gradients