Large-Scale Neural Scene Disentanglement Approach for Self-Driving Simulation

Chunmao Jiang,Runxin Niu,Ziyu Chen,Chen Hua,Xinkai Kuang,Xuhui Fu,Biao Yu,Hui Zhu

DOI: https://doi.org/10.1109/tiv.2024.3390795

IF: 8.2

2024-01-01

IEEE Transactions on Intelligent Vehicles

Abstract:Recent neural radiance field (NeRF) methods have demonstrated the feasibility of closed-loop self-driving vehicle (SDV) simulation by learning accurate view synthesis. However, the fast and high-quality reconstruction of such large outdoor scenes with long and narrow camera trajectories is challenging, as its performance is highly dependent on the neural scene representation, spatial partitioning, camera pose accuracy, etc. Most existing approaches mainly focus on novel view synthesis using RGB images, or add lidar data for depth supervision of the training process. In this paper, we proposed a novel fast neural rendering framework that fully exploits the SDV-specific sensor data. The localization and mapping technique reconstructs a point-cloud map, which is introduced as a geometric prior to supervised space partitioning, reducing wasted capacity and speeding up training. Our framework further extends previous large-scale neural rendering methods with a superior efficient self-driving scene representation strategy, i.e., multiple proximal view spaces plus a shared distant space, which mitigates the performance degradation caused by unbalanced capacity allocation on the long and narrow SDV trajectories. Extensive experiments on the collected large-scale datasets and public outdoor datasets demonstrate that the proposed neural rendering framework is capable of rendering higher quality images of challenging self-driving views to the comparable training time of other fast NeRF methods.