Physically Based Differentiable Rendering： a Survey
Xing Jiankai,Xu Kun
DOI: https://doi.org/10.11834/jig.230715
2024-01-01
Abstract:Rendering has been a prominent subject in the field of computer graphics for an extended period.It can be regarded as a function that accepts an abstract scene description as input and typically generates a 2D image as output.The theory and practice of rendering have remarkably advanced through years of research.In recent years,inverse rendering has emerged as a new research focus in the field of computer graphics due to the development of digital technology.The objective of inverse rendering is to reverse the rendering process and deduce scene parameters from the output image,which is equivalent to solving the inverse function of the rendering function.This process plays a crucial role in addressing perception problems in diverse advanced technological domains,including virtual reality,autonomous driving,and robot-ics.Numerous methods exist for implementing inverse rendering,with the current mainstream framework being optimiza-tion through"analysis by synthesis".First,it estimates a set of initial scene parameters,then performs forward rendering on the scene,compares the rendered result with the target image,and then minimizes the difference(loss function)by opti-mizing the scene parameters using gradient descent-based method.This pipeline necessitates the ability to compute the derivatives of the output image in forward rendering with respect to the input parameters.Consequently,differentiable ren-dering has emerged to fulfill this requirement.Specifically,the research topic of differentiable rendering is to convert the forward rendering pipeline in computer graphics into a differentiable form,enabling the differentiation of the output image with respect to input parameters such as geometry,material,light source,and camera.Currently,forward rendering can be broadly categorized into three types:rasterization-based rendering,physically based rendering,and the emerging neural rendering.Rasterization-based rendering is a fundamental technique in computer graphics that converts geometric shapes into pixels for display.It involves projecting 3D objects onto a 2D screen,performing hidden surface removal,shading,and texturing to create realistic images efficiently.While rasterization is fast and suitable for real-time applications,it may lack physical accuracy in simulating light interactions.By contrast,physically based rendering aims to simulate real-world light behavior accurately by considering the physical properties of materials,light sources,and the environment.It calcu-lates how light rays interact with surfaces,accounting for reflections,refractions,and scattering to produce photorealistic visual results.This method prioritizes realism and is widely used in industries,such as animation,gaming,and visual effects.Neural rendering is an emerging rendering technique in recent years,mainly used for image-based rendering tasks.In contrast to traditional graphics rendering,image-based rendering does not require any explicit 3D scene information(geometry,materials,lighting,etc.),but instead implicitly encodes scenes through a sequence of 2D images sampled from different viewpoints,enabling the generation of images of the scene from any viewpoint.Accordingly,differentiable rendering can also be categorized into three types:differentiable rasterization,physically based differentiable rendering,and differentiable neural rendering.In differentiable rasterization,many works employ approximate methods to compute approximate derivatives of the rasterization process for backpropagation of gradients or modify steps in the traditional render-ing pipeline(usually rasterization and testing/blending steps)to make pixels differentiable with respect to vertices.Neural rendering is naturally differentiable because its rendering process is conducted through neural networks.For physically based differentiable rendering,accurately calculating the gradient of the image concerning scene parameters is challenging because of the intricate nature of geometry,material,and light transmission processes.Therefore,this study concentrates on recent research in the field of physically based differentiable rendering.The article is organized into the following sec-tions:Section 1 introduces the computational methods of forward rendering and differentiable rendering from an abstract standpoint and two types of method for correctly computing boundary integral:edge sampling and reparameterization.Sec-tion 2 explores the attainment of differentiable rendering for distinct representations of geometry,such as volumetric repre-sentation,signed distance field,height field,and vectorized geometry;materials,such as volumetric material;parameter-ized bidirectional reflectance distribution function,bidirectional surface scattering reflectance distribution function,and continuously varying refractive index fields;and camera-related parameters,such as pixel reconstruction filter and time-of-flight camera.Section 3 focuses on enhancing the efficiency and robustness of differentiable rendering,including efficient sampling,high-efficiency system and framework,language for differentiable rendering and several techniques,to enhance the robustness of differentiable rendering.Section 4 showcases the application of differentiable rendering in practical tasks,which can be generally divided in three types:single-object reconstruction,object and environment light reconstruc-tion,and scene reconstruction.Section 5 discusses the future development trends of differentiable rendering,including improving efficiency,robustness of differentiable rendering,and combining differentiable rendering with other methods.