Abstract:Physically-based renderings contain Monte-Carlo noise, with variance that increases as the number of rays per pixel decreases. This noise, while zero-mean for good modern renderers, can have heavy tails (most notably, for scenes containing specular or refractive objects). Learned methods for restoring low fidelity renders are highly developed, because suppressing render noise means one can save compute and use fast renders with few rays per pixel. We demonstrate that a diffusion model can denoise low fidelity renders successfully. Furthermore, our method can be conditioned on a variety of natural render information, and this conditioning helps performance. Quantitative experiments show that our method is competitive with SOTA across a range of sampling rates. Qualitative examination of the reconstructions suggests that the image prior applied by a diffusion method strongly favors reconstructions that are like real images -- so have straight shadow boundaries, curved specularities and no fireflies.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the noise problem in Monte Carlo rendering. Specifically, physically - based rendering will introduce Monte Carlo noise, and the variance of this noise increases as the number of rays per pixel decreases. Although modern renderers can produce zero - mean noise, in scenes containing specular or refractive objects, the noise may have a heavy - tailed distribution. This makes the quality of the rendering results at low sampling rates poor.
To solve this problem, the paper proposes a denoising method based on the diffusion model. The main contributions and innovations of this method are as follows:
1. **Utilizing large - scale image - generation foundation models for denoising**: For the first time, the paper applies large - scale pre - trained image - generation foundation models to Monte Carlo rendering denoising and proves that the information provided by the conditional rendering buffer can significantly improve the performance of the diffusion model. To achieve this, the author applies the ControlNet architecture to a diffusion model in pixel space.
2. **Quantitative and qualitative evidence showing superiority over existing methods**: Through a series of experiments, the paper demonstrates the competitive performance of its method compared to the existing state - of - the - art methods at multiple sampling rates and shows better visual effects. In particular, the diffusion model can generate reconstruction results closer to the real image, with straighter shadow boundaries, curved highlight effects, and no "fireflies" phenomenon.
### Detailed Explanation
#### Background and Motivation
- **Monte Carlo noise problem**: The noise in Monte Carlo rendering increases as the number of rays per pixel decreases, especially when dealing with specular or refractive objects, the noise may have a heavy - tailed distribution.
- **Limitations of existing solutions**: Traditional denoising methods either rely on linear regression models and hand - designed filters or use deep - learning methods, but these methods often have poor performance when dealing with complex scenes or require a large amount of computing resources to achieve the desired effect.
#### Method Overview
- **Application of the diffusion model**: The paper proposes a denoising method based on the diffusion model. This model can operate in pixel space and can be provided with additional information through conditional rendering buffers (such as normals, albedo, depth, etc.).
- **ControlNet architecture**: The author introduces a trainable Control Module, similar to the ControlNet architecture, to process various auxiliary feature buffers from the renderer and add its output to the decoder block of the diffusion model.
#### Experimental Results
- **Quantitative evaluation**: Through multiple measurement standards such as L1 error, PSNR, and LPIPS, the paper demonstrates the superior performance of its method at different sampling rates.
- **Qualitative evaluation**: Through comparative experiments, the paper shows that the images generated by its method are more realistic and the details are more reasonable, especially when dealing with shadows, highlights, and undersampled edges.
In conclusion, this paper aims to provide an efficient and high - quality Monte Carlo rendering denoising method by introducing the diffusion model and conditional rendering buffers, thereby reducing computing costs and improving rendering quality.