Abstract:3D Gaussian Splatting (3DGS) has attracted significant attention for its potential to revolutionize 3D representation, rendering, and interaction. Despite the rapid growth of 3DGS research, its direct application to Extended Reality (XR) remains underexplored. Although many studies recognize the potential of 3DGS for XR, few have explicitly focused on or demonstrated its effectiveness within XR environments. In this paper, we aim to synthesize innovations in 3DGS that show specific potential for advancing XR research and development. We conduct a comprehensive review of publicly available 3DGS papers, with a focus on those referencing XR-related concepts. Additionally, we perform an in-depth analysis of innovations explicitly relevant to XR and propose a taxonomy to highlight their significance. Building on these insights, we propose several prospective XR research areas where 3DGS can make promising contributions, yet remain rarely touched. By investigating the intersection of 3DGS and XR, this paper provides a roadmap to push the boundaries of XR using cutting-edge 3DGS techniques.
What problem does this paper attempt to address?
The main problem that this paper attempts to solve is to apply 3D Gaussian Splatting (3DGS) technology more effectively in the Extended Reality (XR) field. Although 3DGS shows great potential in 3D representation, rendering, and interaction, its direct application in the XR environment is still less explored. Specifically, the paper aims to:
1. **Integrate existing innovations**: Through a systematic review of 272 publicly available 3DGS - related literatures, screen out the research contents related to XR, and evaluate their applicability and potential impact on the XR field.
2. **In - depth analysis of specific innovations**: Analyze in detail those 3DGS innovations that are clearly targeted at XR applications, and propose a taxonomy to highlight the importance of these innovations.
3. **Propose future research directions**: Based on the analysis of existing literatures, propose several research areas in which 3DGS has broad prospects but has not been fully developed in XR, providing guidance for future research.
### Specific problems and solutions
- **Real - time performance and interaction challenges**: Traditional methods such as NeRF (Neural Radiance Fields) can achieve realistic scene representation, but have difficulties in real - time performance and integration with XR systems. 3DGS overcomes these problems by explicitly modeling the scene and using parameterized 3D Gaussian distributions, enabling real - time rendering and efficient interaction.
- **Lack of specific application cases**: Although many studies recognize the potential of 3DGS in XR, there are few actual application cases. To this end, the paper summarizes three representative works (VR - GS, DualGS, RGCA), showing how 3DGS is specifically applied in the XR environment, and providing specific implementation and evaluation examples.
- **Construct taxonomy and roadmap**: In order to better understand the application potential of 3DGS in XR, the paper proposes a taxonomy covering five key areas: 3D content creation, rendering and visualization, interaction and manipulation, system optimization and efficiency, and professional applications. Each area details relevant innovations and their impact on XR.
### Future prospects
The paper also explores future research directions for the combination of 3DGS and XR, including but not limited to:
- **Mesh modeling**: Use 3DGS for high - fidelity mesh modeling to create more realistic virtual environments.
- **Dynamic scene representation**: Improve the real - time capture, processing, and rendering of 4D volumetric videos to support large - scale virtual spaces.
- **Open - world scene understanding**: Enhance the scene understanding ability of AR/MR applications through open - vocabulary 3D segmentation.
- **Hand - tracking**: Improve the real - time capture and rendering accuracy of hand movements and gestures.
- **Perspective function**: Enhance the perspective ability of XR devices to make virtual content seamlessly integrate with the real environment.
- **Immersive visualization**: Achieve efficient visualization of complex data sets through 3DGS, and enhance the user interaction experience in immersive XR applications.
In summary, this paper aims to promote the further development of 3DGS in the XR field by integrating existing research results and proposing new research directions, laying the foundation for a more immersive, interactive, and intelligent XR experience.