Material Transforms from Disentangled NeRF Representations

Ivan Lopes,Jean-François Lalonde,Raoul de Charette

2024-11-13

Abstract:In this paper, we first propose a novel method for transferring material transformations across different scenes. Building on disentangled Neural Radiance Field (NeRF) representations, our approach learns to map Bidirectional Reflectance Distribution Functions (BRDF) from pairs of scenes observed in varying conditions, such as dry and wet. The learned transformations can then be applied to unseen scenes with similar materials, therefore effectively rendering the transformation learned with an arbitrary level of intensity. Extensive experiments on synthetic scenes and real-world objects validate the effectiveness of our approach, showing that it can learn various transformations such as wetness, painting, coating, etc. Our results highlight not only the versatility of our method but also its potential for practical applications in computer graphics. We publish our method implementation, along with our synthetic/real datasets on <a class="link-external link-https" href="https://github.com/astra-vision/BRDFTransform" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to transfer material transformations in different scenarios, especially learning from a known material transformation and applying it to new scenarios with similar material properties. Specifically, the author proposes a method based on disentangled NeRF (Neural Radiance Field) representation to achieve cross - scene material property conversion.

### Specific description of the problem

1. **Complexity of material transformation**: Some material transformations (such as applying a layer of varnish) will significantly change the appearance of the scene, making it complex and time - consuming to recapture the transformed scene.
2. **Limitations of existing methods**: Current methods usually need to recapture the transformed scene, which is very difficult in practice, especially when there are a large variety of transformations (such as wetting, dusting, painting, etc.).
3. **Requirement for cross - scene application**: Researchers hope to be able to transfer a material transformation from one scene to another and be able to control the degree of transformation.

### Core objectives of the paper

- **Learning material transformation**: Learn the changes in material properties by observing images of the same scene under two different conditions (for example, the original state and the transformed state).
- **Cross - scene application**: Apply the learned material transformation to other new scenes with similar materials, thereby predicting the appearance of these new scenes under the same transformation.
- **Controlling transformation intensity**: Be able to control the intensity when applying the transformation to achieve different degrees of effect.

### Method overview

The author proposes a joint optimization framework, using the disentangled NeRF representation to learn the mapping function \( F \) of material properties, which can map the material properties \( \beta_0 \) of the original scene to the material properties \( \beta_1 \) of the transformed scene. The specific steps are as follows:

1. **Data collection**: Obtain multi - view images of the same scene in the original state and the transformed state.
2. **Joint optimization**: Simultaneously optimize the geometric structure and material properties of the original scene and the transformed scene through the disentangled NeRF representation.
3. **Learning the mapping function**: Use a small MLP network \( F \) to learn the mapping relationship between material properties.
4. **Cross - scene application**: Apply the learned mapping function \( F \) to new scenes to achieve the transfer of material transformation.

### Experimental verification

The author verifies the effectiveness of the method through synthetic data sets and real - world data sets. The experimental results show that this method can not only accurately learn material transformations, but also faithfully reproduce these transformations in new scenes, while maintaining the decomposition accuracy of geometric structure and material properties.

In conclusion, this paper aims to solve the problem of how to efficiently and accurately learn and transfer material transformations, providing a new solution for the field of computer graphics.