Abstract:Rendering bridges the gap between 2D vision and 3D scenes by simulating the physical process of image formation. By inverting such renderer, one can think of a learning approach to infer 3D information from 2D images. However, standard graphics renderers involve a fundamental step called rasterization, which prevents rendering to be differentiable. Unlike the state-of-the-art differentiable renderers (Kato et al. 2018 and Loper 2018), which only approximate the rendering gradient in the backpropagation, we propose a natually differentiable rendering framework that is able to (1) directly render colorized mesh using differentiable functions and (2) back-propagate efficient supervisions to mesh vertices and their attributes from various forms of image representations. The key to our framework is a novel formulation that views rendering as an aggregation function that fuses the probabilistic contributions of all mesh triangles with respect to the rendered pixels. Such formulation enables our framework to flow gradients to the occluded and distant vertices, which cannot be achieved by the previous state-of-the-arts. We show that by using the proposed renderer, one can achieve significant improvement in 3D unsupervised single-view reconstruction both qualitatively and quantitatively. Experiments also demonstrate that our approach can handle the challenging tasks in image-based shape fitting, which remain nontrivial to existing differentiable renders.

Efficient Graphics Representation with Differentiable Indirection

DIST: Rendering Deep Implicit Signed Distance Function with Differentiable Sphere Tracing

Differentiable Rendering of Parametric Geometry

A Simple Approach to Differentiable Rendering of SDFs

Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer

Differentiable Shadow Mapping for Efficient Inverse Graphics

DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer

Differentiable Direct Volume Rendering

A General Differentiable Mesh Renderer for Image-Based 3D Reasoning

Modular Primitives for High-Performance Differentiable Rendering

Differentiable Rendering Using RGBXY Derivatives and Optimal Transport.

Differentiable time-gated rendering

Path-space differentiable rendering of participating media

DFR: Differentiable Function Rendering for Learning 3D Generation from Images

Higher-order Differentiable Rendering

Differentiable Point-based Inverse Rendering

Differentiable Imaging: A New Tool for Computational Optical Imaging

Efficient Multi-View Inverse Rendering Using a Hybrid Differentiable Rendering Method

Analytical Derivatives for Differentiable Renderer: 3D Pose Estimation by Silhouette Consistency

Efficient Differentiation of Pixel Reconstruction Filters for Path-Space Differentiable Rendering

Multirate Shading with Piecewise Interpolatory Approximation