Application of 3D Gaussian Splatting for Cinematic Anatomy on Consumer Class Devices

Simon Niedermayr,Christoph Neuhauser,Kaloian Petkov,Klaus Engel,Rüdiger Westermann

2024-06-25

Abstract:Interactive photorealistic rendering of 3D anatomy is used in medical education to explain the structure of the human body. It is currently restricted to frontal teaching scenarios, where even with a powerful GPU and high-speed access to a large storage device where the data set is hosted, interactive demonstrations can hardly be achieved. We present the use of novel view synthesis via compressed 3D Gaussian Splatting (3DGS) to overcome this restriction, and to even enable students to perform cinematic anatomy on lightweight and mobile devices. Our proposed pipeline first finds a set of camera poses that captures all potentially seen structures in the data. High-quality images are then generated with path tracing and converted into a compact 3DGS representation, consuming < 70 MB even for data sets of multiple GBs. This allows for real-time photorealistic novel view synthesis that recovers structures up to the voxel resolution and is almost indistinguishable from the path-traced images

Graphics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to achieve interactive photo - realistic rendering of 3D anatomy in medical education, especially the application of Cinematic Anatomy (CA) on lightweight and mobile devices. Currently, such interactive demonstrations are mainly limited to front - end teaching scenarios, and it is difficult to achieve real - time interaction even with powerful GPUs and high - speed access storage devices. The paper proposes a novel view - synthesis method based on compressed 3D Gaussian Splatting (3DGS) to overcome this limitation and enable students to perform Cinematic Anatomy on lightweight and mobile devices.

Specifically, the paper addresses the following key issues:

1. **Data Size and Performance Limitations**: Current 3D anatomy applications are limited by the size of the data set and the performance of the device, especially on mobile devices. By compressing the 3DGS representation, the paper significantly reduces memory consumption, enabling fast download under low - bandwidth conditions and real - time rendering on mobile devices with mid - to - low - end GPUs.

2. **High - Quality Rendering**: Although traditional path - tracing methods can generate high - quality images, the computational cost is extremely high and real - time rendering cannot be achieved. The 3DGS method proposed in the paper can achieve a rendering speed increase of more than two orders of magnitude while maintaining an image quality close to that of path - tracing.

3. **New - View Synthesis**: To achieve high - quality new - view synthesis, the paper proposes a processing pipeline that includes automatic camera - view selection, generation of high - quality training images, optimization of the 3DGS representation, and compression of the representation. These steps together ensure efficient rendering of high - quality images from any perspective.

4. **Reconstruction of Transparent Materials**: In volume - rendering applications, it is necessary not only to reconstruct RGB colors but also to reconstruct the accumulated opacity (alpha) of each pixel. The paper extends 3DGS to enable differentiable rendering with an alpha channel, thereby greatly improving the reconstruction quality and eliminating artifacts caused by the background.

Through these methods, the paper successfully achieves interactive photo - realistic 3D - anatomy rendering on lightweight and mobile devices, providing new tools and methods for medical education.