NeMF: Inverse Volume Rendering with Neural Microflake Field

Youjia Zhang,Teng Xu,Junqing Yu,Yuteng Ye,Junle Wang,Yanqing Jing,Jingyi Yu,Wei Yang

2023-04-04

Abstract:Recovering the physical attributes of an object's appearance from its images captured under an unknown illumination is challenging yet essential for photo-realistic rendering. Recent approaches adopt the emerging implicit scene representations and have shown impressive results.However, they unanimously adopt a surface-based representation,and hence can not well handle scenes with very complex geometry, translucent object and etc. In this paper, we propose to conduct inverse volume rendering, in contrast to surface-based, by representing a scene using microflake volume, which assumes the space is filled with infinite small flakes and light reflects or scatters at each spatial location according to microflake distributions. We further adopt the coordinate networks to implicitly encode the microflake volume, and develop a differentiable microflake volume renderer to train the network in an end-to-end way in principle.Our NeMF enables effective recovery of appearance attributes for highly complex geometry and scattering object, enables high-quality relighting, material editing, and especially simulates volume rendering effects, such as scattering, which is infeasible for surface-based approaches.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper primarily addresses the following issues:

1. **Inverse Volume Rendering**:
- Recovering the physical properties of a scene (such as shape, reflectance, and lighting) from multi-view images, especially under unknown natural lighting conditions.
- Existing methods are mostly based on surface representations (such as SDF), which struggle with complex geometries and translucent objects.

2. **High-Fidelity Rendering and Material Editing**:
- Providing high-quality relighting, material editing, and simulating volume scattering effects.
- Compared to traditional surface-based methods, this approach can better handle complex geometries and translucent objects.

3. **Overcoming the Limitations of Existing Methods**:
- Existing inverse rendering methods have shortcomings when dealing with complex geometries and translucent objects, particularly surface-based methods that cannot effectively simulate volume scattering effects.
- This paper proposes a new volume representation method—Neural Microflake Field (NeMF)—to overcome these limitations.

Through these improvements, the paper aims to enhance the quality of inverse rendering and expand its application scenarios.