SAMa: Material-aware 3D Selection and Segmentation

Michael Fischer,Iliyan Georgiev,Thibault Groueix,Vladimir G. Kim,Tobias Ritschel,Valentin Deschaintre

2024-11-29

Abstract:Decomposing 3D assets into material parts is a common task for artists and creators, yet remains a highly manual process. In this work, we introduce Select Any Material (SAMa), a material selection approach for various 3D representations. Building on the recently introduced SAM2 video selection model, we extend its capabilities to the material domain. We leverage the model's cross-view consistency to create a 3D-consistent intermediate material-similarity representation in the form of a point cloud from a sparse set of views. Nearest-neighbour lookups in this similarity cloud allow us to efficiently reconstruct accurate continuous selection masks over objects' surfaces that can be inspected from any view. Our method is multiview-consistent by design, alleviating the need for contrastive learning or feature-field pre-processing, and performs optimization-free selection in seconds. Our approach works on arbitrary 3D representations and outperforms several strong baselines in terms of selection accuracy and multiview consistency. It enables several compelling applications, such as replacing the diffuse-textured materials on a text-to-3D output, or selecting and editing materials on NeRFs and 3D-Gaussians.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to solve the problems of 3D object material selection and segmentation. Specifically, the authors introduce a method named **SAMa (Select Any Material)** for material selection in various 3D representations (such as meshes, NeRFs, 3D Gaussian distributions, etc.). The following are the main problems and goals of the paper:

1. **Existing challenges**:
- Current material understanding work mainly focuses on the 2D image field, while 3D material selection is still a highly manual process.
- Existing 2D material selection models cannot be directly extended to the 3D field because these models lack consistency between different views.
- The 3D field has inherent challenges such as self - occlusion and view - dependent effects, and it is necessary to accurately propagate model predictions to new, unseen views.

2. **Paper goals**:
- **Multi - view consistency**: Ensure that the results are consistent when performing material selection from any view.
- **Fast and efficient selection**: Achieve interactive material selection at the second - level, avoiding long - term optimization processes.
- **Support for multiple 3D representations**: Applicable to any 3D representation form that can be rendered into an image and query depth information.
- **Downstream applications**: Provide multiple application scenarios, such as replacing the diffuse texture material in text - to - 3D output, editing materials in NeRFs and 3D Gaussian distributions, etc.

3. **Solutions**:
- **Video - based selection model**: Draw on the recently proposed SAM2 video selection model and fine - tune it to adapt to the material selection task.
- **Video dataset with fine - grained annotations**: Construct a video dataset with dense per - pixel material annotations to train the model to have cross - frame consistency.
- **Lightweight 2D to 3D promotion**: Integrate 2D similarity maps of multiple views into a 3D similarity point cloud to achieve efficient 3D selection.

### Core contributions of the paper

- **Adaptation of video object selection model**: Apply the SAM2 model to 3D material selection by fine - tuning on the newly designed rendered video dataset.
- **Fast and efficient 3D projection**: Utilize cross - frame consistency to achieve fast and efficient 3D material selection.
- **Multi - modal support**: Support segmentation and editing of multiple 3D representations such as meshes, radiance fields, and 3D Gaussian distributions.

Through these methods, the paper significantly improves the accuracy and multi - view consistency of 3D material selection and shows its potential in multiple application scenarios.