Abstract:Simulation models for perception sensors are integral components of automotive simulators used for the virtual Verification and Validation (V\&V) of Autonomous Driving Systems (ADS). These models also serve as powerful tools for generating synthetic datasets to train deep learning-based perception models. Lidar is a widely used sensor type among the perception sensors for ADS due to its high precision in 3D environment scanning. However, developing realistic Lidar simulation models is a significant technical challenge. In particular, unrealistic models can result in a large gap between the synthesised and real-world point clouds, limiting their effectiveness in ADS applications. Recently, deep generative models have emerged as promising solutions to synthesise realistic sensory data. However, for Lidar simulation, deep generative models have been primarily hybridised with conventional algorithms, leaving unified generative approaches largely unexplored in the literature. Motivated by this research gap, we propose a unified generative framework to enhance Lidar simulation fidelity. Our proposed framework projects Lidar point clouds into depth-reflectance images via a lossless transformation, and employs our novel Controllable Lidar point cloud Generative model, CoLiGen, to translate the images. We extensively evaluate our CoLiGen model, comparing it with the state-of-the-art image-to-image translation models using various metrics to assess the realness, faithfulness, and performance of a downstream perception model. Our results show that CoLiGen exhibits superior performance across most metrics. The dataset and source code for this research are available at <a class="link-external link-https" href="https://github.com/hamedhaghighi/CoLiGen.git" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the problem of the realism of Lidar simulation in Autonomous Driving Systems (ADS). Specifically, the goals of the paper are: 1. **Narrow the Sim - to - Real Gap**: - Lidar, as a commonly used perception sensor, plays an important role in autonomous driving systems because it can scan the 3D environment with high precision. - However, developing a realistic Lidar simulation model is a major technical challenge. Unrealistic simulation models will lead to significant differences between synthetic point clouds and real - world point clouds, thus limiting their effectiveness in ADS applications. 2. **Propose a unified generation framework to improve the realism of Lidar simulation**: - Currently, deep generative models have been used to synthesize realistic sensory data, but in Lidar simulation, these models are usually used in combination with traditional algorithms and lack a unified generation method. - The paper proposes a new controllable generative model named CoLiGen, which can translate all Lidar data attributes (including shape features and reflectivity characteristics) into more realistic representations, thereby improving the overall simulation fidelity. 3. **Evaluate the performance of the generative model**: - The paper extensively evaluates the CoLiGen model through multiple metrics to measure its realism, fidelity, and the performance of downstream perception models. - The results show that CoLiGen outperforms the existing state - of - the - art image - to - image translation models on most evaluation metrics. ### Key contributions - **Propose the CoLiGen model**: This is the first controllable generative model that can translate all key Lidar data attributes, including shape features and reflectivity characteristics. - **Comprehensive evaluation**: CoLiGen has been evaluated from multiple perspectives such as realism, fidelity, and the performance of downstream perception models. - **Introduce a new dataset**: A new dataset named semantic - CARLA has been created, which contains synthetic, point - by - point labeled Lidar point clouds and RGB images for training and evaluating the model. - **Technical innovation**: Innovative techniques such as decomposed raydrop synthesis and contrastive learning are adopted to improve the quality and consistency of synthetic point clouds. Through these efforts, the paper provides new solutions for improving the realism and reliability of Lidar simulation, which helps to accelerate the virtual verification and validation (V&V) process of autonomous driving systems.

A Unified Generative Framework for Realistic Lidar Simulation in Autonomous Driving Systems

LidarDM: Generative LiDAR Simulation in a Generated World

GAN-Based LiDAR Intensity Simulation

Learning to Simulate Realistic LiDARs

LiDAR Sensor modeling and Data augmentation with GANs for Autonomous driving

A Cgans-Based Scene Reconstruction Model Using Lidar Point Cloud

Review of the Learning-based Camera and Lidar Simulation Methods for Autonomous Driving Systems

A Shared Representation for Photorealistic Driving Simulators

GAN-Based LiDAR Translation between Sunny and Adverse Weather for Autonomous Driving and Driving Simulation

PCGen: Point Cloud Generator for LiDAR Simulation

A LiDAR Point Cloud Generator: from a Virtual World to Autonomous Driving

Exploring Generative AI for Sim2Real in Driving Data Synthesis

Data Generation Using Simulation Technology to Improve Perception Mechanism of Autonomous Vehicles

Taming Transformers for Realistic Lidar Point Cloud Generation

LiDAR-as-Camera for End-to-End Driving

LiDAR View Synthesis for Robust Vehicle Navigation Without Expert Labels

Generating Realistic Simulation of LiDAR Data Based on Scene Flow for Autonomous Vehicles Testing

Study of a Multi-Beam LiDAR Perception Assessment Model for Real-Time Autonomous Driving

OLiDM: Object-aware LiDAR Diffusion Models for Autonomous Driving

Towards Realistic Scene Generation with LiDAR Diffusion Models

UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generation