Abstract:Physically based differentiable rendering has become an increasingly important topic in recent years. A common pipeline computes local color derivatives of light paths or pixels with respect to arbitrary scene parameters, and enables optimizing or recovering the scene parameters through iterative gradient descent by minimizing the difference between rendered and target images. However, existing approaches cannot robustly handle complex illumination effects including reflections, refractions, caustics, shadows, and highlights, especially when the initial and target locations of such illumination effects are not close to each other in the image space. To address this problem, we propose a novel data structure named extended path space manifolds. The manifolds are defined in the combined space of path vertices and scene parameters. By enforcing geometric constraints, the path vertices could be implicitly and uniquely determined by perturbed scene parameters. This enables the manifold to track specific illumination effects and the corresponding paths, i.e., specular paths will still be specular paths after scene parameters are perturbed. Besides, the path derivatives with respect to scene parameters could be computed by solving small linear systems. We further propose a physically based differentiable rendering method built upon the theoretical results of extended path space manifolds. By incorporating the path derivatives computed from the manifolds and an optimal transport based loss function, our method is demonstrated to be more effective and robust than state-of-the-art approaches in inverse rendering applications involving complex illumination effects.

Residual path integrals for re-rendering

Efficient screen space approach for high complex dynamic scenes in real-time rendering

Path-space differentiable rendering of participating media

A Practical Path Guiding Method for Participating Media

Efficient Differentiation of Pixel Reconstruction Filters for Path-Space Differentiable Rendering

Path‐space Motion Estimation and Decomposition for Robust Animation Filtering

Path Verification for Dynamic Indirect Illumination

Parallel and Adaptive Visibility Sampling for Rendering Dynamic Scenes with Spatially Varying Reflectance.

Extended Path Space Manifolds for Physically Based Differentiable Rendering.

Rendering Participating Media Using Path Graphs

Amortizing Samples in Physics-Based Inverse Rendering Using ReSTIR

Efficient Caustics Rendering Via Spatial and Temporal Path Reuse

Image and Video Retexturing

Accelerating Inverse Rendering By Using a GPU and Reuse of Light Paths

A New Radiosity Approach by Procedural Refinements for Realistic Image Sythesis

World‐Space Spatiotemporal Path Resampling for Path Tracing

Hybrid Forward Resampling and Volume Rendering

Higher-order Differentiable Rendering

A Matrix Sampling-And-Recovery Approach for Many-Lights Rendering

Efficient Spatial Resampling Using the PDF Similarity

MIRReS: Multi-bounce Inverse Rendering using Reservoir Sampling