3D Gaussian Splatting in Robotics: A Survey

Siting Zhu,Guangming Wang,Dezhi Kong,Hesheng Wang

2024-10-16

Abstract:Dense 3D representations of the environment have been a long-term goal in the robotics field. While previous Neural Radiance Fields (NeRF) representation have been prevalent for its implicit, coordinate-based model, the recent emergence of 3D Gaussian Splatting (3DGS) has demonstrated remarkable potential in its explicit radiance field representation. By leveraging 3D Gaussian primitives for explicit scene representation and enabling differentiable rendering, 3DGS has shown significant advantages over other radiance fields in real-time rendering and photo-realistic performance, which is beneficial for robotic applications. In this survey, we provide a comprehensive understanding of 3DGS in the field of robotics. We divide our discussion of the related works into two main categories: the application of 3DGS and the advancements in 3DGS techniques. In the application section, we explore how 3DGS has been utilized in various robotics tasks from scene understanding and interaction perspectives. The advance of 3DGS section focuses on the improvements of 3DGS own properties in its adaptability and efficiency, aiming to enhance its performance in robotics. We then summarize the most commonly used datasets and evaluation metrics in robotics. Finally, we identify the challenges and limitations of current 3DGS methods and discuss the future development of 3DGS in robotics.

Robotics

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to explore the application and technological advancements of 3D Gaussian Splatting (3DGS) in the field of robotics. Specifically, the paper addresses the following issues:

1. **Environment Representation**:
- **Explicit Representation**: Traditional Neural Radiance Fields (NeRF) perform well in implicit coordinate representation but suffer from low efficiency in per-pixel ray casting rendering. 3DGS improves real-time rendering and photo-realistic performance through explicit Gaussian splatting representation, making it more suitable for robotic applications.
- **Scene Modeling**: 3DGS models the environment using a set of 3D Gaussian primitives with learnable parameters, providing an explicit representation of the scene.

2. **Application Scenarios**:
- **Scene Understanding**: The paper discusses how 3DGS can be used in robotic tasks for scene understanding, including static and dynamic scene reconstruction, scene segmentation, and editing.
- **Interactive Tasks**: The application of 3DGS in robot-environment interaction, such as manipulation, navigation, localization, and path planning.

3. **Technological Advancements**:
- **Adaptability and Efficiency**: The paper explores how to improve the adaptability and efficiency of 3DGS to enhance its performance in robotic tasks.
- **Optimization Methods**: Introduces optimization methods for 3DGS, including loss function design and adaptive density control techniques.

4. **Datasets and Evaluation Metrics**:
- **Common Datasets**: Summarizes commonly used 3DGS datasets in the field of robotics.
- **Evaluation Metrics**: Discusses common metrics used to evaluate the performance of 3DGS.

5. **Challenges and Future Directions**:
- **Current Limitations**: Identifies the challenges and limitations of existing 3DGS methods.
- **Future Research**: Discusses future development directions of 3DGS in the field of robotics, especially in the application of large-scale dynamic scene reconstruction.

Overall, this paper provides a comprehensive review of the application and technological advancements of 3DGS in the field of robotics, offering researchers and engineers a reference for in-depth understanding and application of 3DGS.