Abstract:We present SGS-SLAM, the first semantic visual SLAM system based on Gaussian Splatting. It incorporates appearance, geometry, and semantic features through multi-channel optimization, addressing the oversmoothing limitations of neural implicit SLAM systems in high-quality rendering, scene understanding, and object-level geometry. We introduce a unique semantic feature loss that effectively compensates for the shortcomings of traditional depth and color losses in object optimization. Through a semantic-guided keyframe selection strategy, we prevent erroneous reconstructions caused by cumulative errors. Extensive experiments demonstrate that SGS-SLAM delivers state-of-the-art performance in camera pose estimation, map reconstruction, precise semantic segmentation, and object-level geometric accuracy, while ensuring real-time rendering capabilities.
What problem does this paper attempt to address?
### Problems the paper attempts to solve
The paper aims to solve the over - smoothing problem in high - quality rendering, scene understanding and object - level geometric representation in traditional neural implicit SLAM systems. Specifically, the paper proposes SGS - SLAM (Semantic Gaussian Splatting for Neural Dense SLAM), which is a semantic visual SLAM system based on Gaussian splatting. Through multi - channel optimization, this system combines appearance, geometric and semantic features, effectively compensating for the deficiencies of traditional depth and color losses in object optimization.
### Main problems and solutions
1. **Over - smoothing problem**:
- **Problem**: Traditional neural implicit SLAM systems (such as NeRF - based methods) have an over - smoothing problem at object edges, resulting in a lack of fine - grained details in the map and making it difficult to segment, edit and manipulate objects in the scene.
- **Solution**: SGS - SLAM avoids the over - smoothing problem by introducing an explicit Gaussian splatting representation. Gaussian splatting can directly capture complex textures and geometric details, providing high - fidelity reconstruction results.
2. **Large - scale scene processing**:
- **Problem**: NeRF - based methods are prone to catastrophic forgetting when processing large - scale scenes, that is, adding new scenes will affect the accuracy of the previously learned model.
- **Solution**: SGS - SLAM can dynamically add new parameters through explicit Gaussian representation without affecting the existing scene representation, thus solving the problem of large - scale scene processing.
3. **Computational efficiency**:
- **Problem**: NeRF - based methods have low computational efficiency and need to model the entire scene, resulting in complex and time - consuming model tuning.
- **Solution**: SGS - SLAM utilizes the fast rasterization ability of Gaussian splatting to achieve efficient real - time rendering and optimization. The direct projection ability of Gaussian splatting simplifies the parameter optimization process and improves computational efficiency.
4. **Utilization of semantic information**:
- **Problem**: Existing SLAM systems cannot fully utilize semantic information during map construction, resulting in low accuracy in scene understanding and object recognition.
- **Solution**: SGS - SLAM incorporates semantic information into the Gaussian splatting representation. Through a multi - channel optimization strategy, it simultaneously optimizes appearance, geometric and semantic signals, improving the accuracy of camera tracking and scene reconstruction. In addition, semantic information is also used for key - frame selection, further improving the quality of map reconstruction.
### Experimental verification
The paper verifies the superior performance of SGS - SLAM in camera pose estimation, map reconstruction, accurate semantic segmentation and object - level geometric representation through experiments on synthetic datasets (such as Replica) and real - world datasets (such as ScanNet). The experimental results show that SGS - SLAM significantly outperforms existing NeRF - based and 3D Gaussian splatting SLAM systems on multiple metrics.
### Summary
SGS - SLAM effectively solves the over - smoothing problem, the large - scale scene processing problem and the low computational efficiency problem in traditional neural implicit SLAM systems by introducing an explicit Gaussian splatting representation and a multi - channel optimization strategy. At the same time, by fusing semantic information, SGS - SLAM performs well in scene understanding and object manipulation, providing a solid foundation for robotics and mixed - reality applications.