Open-Vocabulary High-Resolution 3D (OVHR3D) Data Segmentation and Annotation Framework

Jiuyi Xu,Meida Chen,Andrew Feng,Yangming Shi,Zifan Yu

2024-12-09

Abstract:In the domain of the U.S. Army modeling and simulation, the availability of high quality annotated 3D data is pivotal to creating virtual environments for training and simulations. Traditional methodologies for 3D semantic and instance segmentation, such as KpConv, RandLA, Mask3D, etc., are designed to train on extensive labeled datasets to obtain satisfactory performance in practical tasks. This requirement presents a significant challenge, given the inherent scarcity of manually annotated 3D datasets, particularly for the military use cases. Recognizing this gap, our previous research leverages the One World Terrain data repository manually annotated databases, as showcased at IITSEC 2019 and 2021, to enrich the training dataset for deep learning models. However, collecting and annotating large scale 3D data for specific tasks remains costly and inefficient. To this end, the objective of this research is to design and develop a comprehensive and efficient framework for 3D segmentation tasks to assist in 3D data annotation. This framework integrates Grounding DINO and Segment anything Model, augmented by an enhancement in 2D image rendering via 3D mesh. Furthermore, the authors have also developed a user friendly interface that facilitates the 3D annotation process, offering intuitive visualization of rendered images and the 3D point cloud.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to solve the problem of the scarcity of high - quality labeled 3D data in the field of United States Army Modeling and Simulation (M&S). Specifically, traditional 3D semantic/instance segmentation methods (such as KpConv, RandLA, Mask3D, etc.) rely on large - scale labeled datasets for training to achieve satisfactory performance. However, manually labeling large - scale 3D datasets is costly and inefficient, especially in military application scenarios.

To address this challenge, the authors propose an Open - Vocabulary High - Resolution 3D (OVHR3D) data segmentation and annotation framework. This framework integrates Grounding DINO (GDINO) and Segment - Anything Model (SAM), and uses enhanced 2D image rendering techniques to assist in the segmentation and annotation of 3D point clouds. In addition, the authors also develop a user - friendly interface (UI) to provide intuitive visualization tools to help users complete 3D data annotation tasks more efficiently.

### Main objectives

1. **Reduce annotation cost and time**: Use automated and intelligent methods to reduce the time and cost of manual annotation.
2. **Improve segmentation accuracy**: Ensure that the segmentation accuracy is not sacrificed while improving efficiency.
3. **Adapt to multiple scenarios**: Be suitable for 3D data annotation in indoor and outdoor environments, including data collected by unmanned aerial vehicles (UAVs), NavVis VLX, and RGB - D cameras.

### Experimental verification

To evaluate the effectiveness of this framework, the authors conducted comparative experiments using indoor scenes from the USC - ICT office and outdoor scenes collected by unmanned aerial vehicles. The results show that this framework is superior to traditional manual annotation methods in terms of efficiency and accuracy, significantly reducing the annotation time and maintaining high segmentation accuracy.

### Summary

This research proposes an innovative OVHR3D data segmentation and annotation framework, aiming to simplify the 3D data annotation process by combining advanced deep - learning models and techniques, thereby promoting the training and application of more complex 3D environmental understanding models.