Robust Roadside Perception: an Automated Data Synthesis Pipeline Minimizing Human Annotation

Rusheng Zhang,Depu Meng,Lance Bassett,Shengyin Shen,Zhengxia Zou,Henry X. Liu

2024-02-09

Abstract:Recently, advancements in vehicle-to-infrastructure communication technologies have elevated the significance of infrastructure-based roadside perception systems for cooperative driving. This paper delves into one of its most pivotal challenges: data insufficiency. The lacking of high-quality labeled roadside sensor data with high diversity leads to low robustness, and low transfer-ability of current roadside perception systems. In this paper, a novel solution is proposed to address this problem that creates synthesized training data using Augmented Reality. A Generative Adversarial Network is then applied to enhance the reality further, that produces a photo-realistic synthesized dataset that is capable of training or fine-tuning a roadside perception detector which is robust to different weather and lighting conditions. Our approach was rigorously tested at two key intersections in Michigan, USA: the Mcity intersection and the State St./Ellsworth Rd roundabout. The Mcity intersection is located within the Mcity test field, a controlled testing environment. In contrast, the State St./Ellsworth Rd intersection is a bustling roundabout notorious for its high traffic flow and a significant number of accidents annually. Experimental results demonstrate that detectors trained solely on synthesized data exhibit commendable performance across all conditions. Furthermore, when integrated with labeled data, the synthesized data can notably bolster the performance of pre-existing detectors, especially in adverse conditions.

Computer Vision and Pattern Recognition,Robotics,Image and Video Processing

What problem does this paper attempt to address?

The paper aims to address the roadside perception problem in infrastructure-assisted autonomous driving systems, specifically the issue of data insufficiency. Current roadside perception systems suffer from poor robustness and generalization ability due to the lack of high-quality, diverse annotated sensor data. The paper proposes an automated data synthesis pipeline based on Augmented Reality (AR) and Generative Adversarial Networks (GAN) to generate realistic synthetic training data. This approach can significantly improve the performance of detectors under different weather and lighting conditions.

Specific contributions include:
1. Proposing an AR rendering pipeline for roadside perception systems, including camera pose estimation, vehicle position and orientation simulation, and AR rendering to generate physically realistic images and their annotation information.
2. Introducing a GAN-based reality enhancement strategy to convert physically realistic images into photo-realistic images.
3. Conducting extensive field evaluations of the model under different lighting and weather conditions, validating the effectiveness of the method in practical applications and the feasibility of large-scale deployment.

In this way, the researchers hope to address the data insufficiency problem in current roadside perception systems, thereby enhancing their performance in complex environments.