Abstract:Gradient-based methods are becoming increasingly important for computer graphics, machine learning, and computer vision. The ability to compute gradients is crucial to optimization, inverse problems, and deep learning. In rendering, the gradient is required with respect to variables such as camera parameters, light sources, scene geometry, or material appearance. However, computing the gradient of rendering is challenging because the rendering integral includes visibility terms that are not differentiable. Previous work on differentiable rendering has focused on approximate solutions. They often do not handle secondary effects such as shadows or global illumination, or they do not provide the gradient with respect to variables other than pixel coordinates. We introduce a general-purpose differentiable ray tracer, which, to our knowledge, is the first comprehensive solution that is able to compute derivatives of scalar functions over a rendered image with respect to arbitrary scene parameters such as camera pose, scene geometry, materials, and lighting parameters. The key to our method is a novel edge sampling algorithm that directly samples the Dirac delta functions introduced by the derivatives of the discontinuous integrand. We also develop efficient importance sampling methods based on spatial hierarchies. Our method can generate gradients in times running from seconds to minutes depending on scene complexity and desired precision. We interface our differentiable ray tracer with the deep learning library PyTorch and show prototype applications in inverse rendering and the generation of adversarial examples for neural networks.

Learning to Rasterize Differentiably

Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction

Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer

DiffCSG: Differentiable CSG via Rasterization

Differentiable Rendering Using RGBXY Derivatives and Optimal Transport.

DFR: Differentiable Function Rendering for Learning 3D Generation from Images

Transforming a Non-Differentiable Rasterizer into a Differentiable One with Stochastic Gradient Estimation

Rasterized Edge Gradients: Handling Discontinuities Differentiably

DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer

Multirate Shading with Piecewise Interpolatory Approximation

Differentiable Rendering of Neural SDFs through Reparameterization

Adaptive Sampling and Reconstruction for Gradient-Domain Rendering

Higher-order Differentiable Rendering

Efficient Differentiation of Pixel Reconstruction Filters for Path-Space Differentiable Rendering

A Simple Approach to Differentiable Rendering of SDFs

Differentiable Rendering for Synthetic Aperture Radar Imagery

DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing

Differentiable Rendering as a Way to Program Cable-Driven Soft Robots

A General Differentiable Mesh Renderer for Image-Based 3D Reasoning

Approximate Differentiable Rendering with Algebraic Surfaces

Differentiable Monte Carlo ray tracing through edge sampling