Abstract:Computing the gradients of a rendering process is paramount for diverse applications in computer vision and graphics. However, accurate computation of these gradients is challenging due to discontinuities and rendering approximations, particularly for surface-based representations and rasterization-based rendering. We present a novel method for computing gradients at visibility discontinuities for rasterization-based differentiable renderers. Our method elegantly simplifies the traditionally complex problem through a carefully designed approximation strategy, allowing for a straightforward, effective, and performant solution. We introduce a novel concept of micro-edges, which allows us to treat the rasterized images as outcomes of a differentiable, continuous process aligned with the inherently non-differentiable, discrete-pixel rasterization. This technique eliminates the necessity for rendering approximations or other modifications to the forward pass, preserving the integrity of the rendered image, which makes it applicable to rasterized masks, depth, and normals images where filtering is prohibitive. Utilizing micro-edges simplifies gradient interpretation at discontinuities and enables handling of geometry intersections, offering an advantage over the prior art. We showcase our method in dynamic human head scene reconstruction, demonstrating effective handling of camera images and segmentation masks.

What problem does this paper attempt to address?

The paper aims to address the problem of gradient computation in the rendering process within computer graphics and computer vision, particularly in handling discontinuities during rasterization. Specifically: 1. **Problem Background**: Computing accurate gradients during rendering is highly challenging, especially in surface-based representations and rasterization rendering. The presence of discontinuities and rendering approximations makes this problem even more complex. 2. **Main Contribution**: The authors propose a new method to compute gradients at visibility discontinuities in rasterization renderers. This method simplifies traditionally complex problems through carefully designed approximation strategies, resulting in a simple, effective, and high-performance solution. They introduce a novel concept—micro-edges, which allows rasterized images to be viewed as the result of a continuous differentiable process, consistent with the inherently non-differentiable discrete pixel rasterization. This eliminates the need for rendering approximations or other forward pass modifications, preserving the integrity of the rendered image. 3. **Specific Application Scenarios**: This technique is applicable to scenarios such as rasterized masks, depth maps, and normal maps, where filtering is not possible. The use of micro-edges simplifies gradient interpretation at discontinuities and can handle geometric intersections, offering advantages over existing methods. 4. **Experimental Validation**: The authors demonstrate the effectiveness of this method in dynamic head scene reconstruction, effectively handling camera images and segmentation masks. In summary, this paper aims to provide a simple, accurate, and efficient method for rasterization gradient computation by introducing the concept of micro-edges, addressing the gradient computation errors present in existing methods when dealing with triangle intersections.

Rasterized Edge Gradients: Handling Discontinuities Differentiably

Image Copy-and-paste with Optimized Gradient

Differentiable Monte Carlo ray tracing through edge sampling

Transforming a Non-Differentiable Rasterizer into a Differentiable One with Stochastic Gradient Estimation

Adaptive Sampling and Reconstruction for Gradient-Domain Rendering

A Novel Robust Color Gradient Estimator for Photographic Volume Visualization

Edge-Aware Volume Smoothing Using L0 Gradient Minimization

A Simple Approach to Differentiable Rendering of SDFs

Learning to Rasterize Differentiably

Robust Color Gradient Estimation for Photographic Volumes.

Deep convolutional reconstruction for gradient-domain rendering

Gradientshop: A perceptually-motivated optimization-framework for image and video processing

DiffCSG: Differentiable CSG via Rasterization

Volumetrically Consistent 3D Gaussian Rasterization

Efficient Road Specular Reflection Removal Based on Gradient Properties

Filtering-Based Reconstruction for Gradient-Domain Rendering

Efficient Differentiation of Pixel Reconstruction Filters for Path-Space Differentiable Rendering

3D Gradient Enhancement

Discontinuity-preserving Normal Integration with Auxiliary Edges

Simplicity Driven Edge Refinement and Color Reconstruction in Image Vectorization.