LiDAR-RT: Gaussian-based Ray Tracing for Dynamic LiDAR Re-simulation

Chenxu Zhou,Lvchang Fu,Sida Peng,Yunzhi Yan,Zhanhua Zhang,Yong Chen,Jiazhi Xia,Xiaowei Zhou

2024-12-20

Abstract:This paper targets the challenge of real-time LiDAR re-simulation in dynamic driving scenarios. Recent approaches utilize neural radiance fields combined with the physical modeling of LiDAR sensors to achieve high-fidelity re-simulation results. Unfortunately, these methods face limitations due to high computational demands in large-scale scenes and cannot perform real-time LiDAR rendering. To overcome these constraints, we propose LiDAR-RT, a novel framework that supports real-time, physically accurate LiDAR re-simulation for driving scenes. Our primary contribution is the development of an efficient and effective rendering pipeline, which integrates Gaussian primitives and hardware-accelerated ray tracing technology. Specifically, we model the physical properties of LiDAR sensors using Gaussian primitives with learnable parameters and incorporate scene graphs to handle scene dynamics. Building upon this scene representation, our framework first constructs a bounding volume hierarchy (BVH), then casts rays for each pixel and generates novel LiDAR views through a differentiable rendering algorithm. Importantly, our framework supports realistic rendering with flexible scene editing operations and various sensor configurations. Extensive experiments across multiple public benchmarks demonstrate that our method outperforms state-of-the-art methods in terms of rendering quality and efficiency. Our project page is at <a class="link-external link-https" href="https://zju3dv.github.io/lidar-rt" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Machine Learning,Robotics

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to achieve real - time re - simulation of LiDAR views in dynamic driving scenarios. Specifically, existing methods face high computational requirements when dealing with large - scale scenarios and cannot achieve real - time LiDAR rendering. In addition, these methods have limitations in modeling and simulating the physical characteristics of LiDAR sensors in complex dynamic scenarios.

### Background and Problems of the Paper

1. **Limitations of Existing Methods**:
- Traditional methods reconstruct 3D scenes using explicit representations (such as dense point clouds or triangular meshes) and generate new LiDAR views through ray casting. Although these methods can produce acceptable results, they are limited by geometric inaccuracies and the inability to model the physical characteristics of LiDAR sensors, and can only handle static scenes.
- Methods based on Neural Radiance Fields (NeRF) combine the rendering capabilities of neural fields and the physical modeling process of LiDAR sensors to achieve physically realistic LiDAR rendering. However, these methods are computationally expensive during training and rendering processes and are difficult to handle complex dynamic scenarios, especially in cases of long - distance vehicle movement and occlusion.

2. **Research Objectives**:
- Propose a new framework, LiDAR - RT, for new - type LiDAR view synthesis in dynamic driving scenarios.
- Achieve efficient, physically accurate LiDAR re - simulation, supporting real - time rendering and flexible scene - editing operations.
- Solve the problems of high computational requirements and inefficiency of existing methods when dealing with large - scale dynamic scenarios.

### Main Contributions

1. **Efficient Rendering Pipeline**: Propose an efficient rendering pipeline based on Gaussian primitives and hardware - accelerated ray - tracing techniques.
2. **Dynamic Scene Modeling**: Introduce scene graphs to handle scene dynamic changes and enhance the modeling ability under various environmental conditions.
3. **Physically Accurate Ray - Tracing**: Design a differentiable ray - tracing framework based on Gaussian primitives to simulate the physical imaging process of LiDAR sensors.
4. **Flexibility and Adaptability**: Support flexible LiDAR sensor configurations and scene - editing operations, which are suitable for multiple downstream tasks.

### Method Overview

- **Dynamic Scene Representation**: Decompose the dynamic scene into a static background and multiple moving objects, and represent each object with a set of Gaussian primitives. Introduce learnable parameters (such as reflection intensity ζ and ray - loss probability β) to simulate the internal characteristics of LiDAR sensors.
- **Gaussian - Primitive - Based Ray - Tracing**: Construct proxy geometries and use ray - tracing techniques for intersection tests, calculate LiDAR properties and accumulate these properties along the rays.
- **Differentiable LiDAR Rendering**: Implement the back - propagation of ray - tracing to support the optimization of scene representation.

Through these improvements, LiDAR - RT can not only achieve high - fidelity LiDAR point - cloud reconstruction and real - time rendering in dynamic driving scenarios, but also support flexible LiDAR sensor operations. Experimental results show that this method outperforms existing methods on multiple publicly available benchmark datasets, especially in terms of rendering quality and efficiency.