Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review

Shanliang Yao,Runwei Guan,Xiaoyu Huang,Zhuoxiao Li,Xiangyu Sha,Yong Yue,Eng Gee Lim,Hyungjoon Seo,Ka Lok Man,Xiaohui Zhu,Yutao Yue

DOI: https://doi.org/10.1109/TIV.2023.3307157

2023-08-23

Abstract:Driven by deep learning techniques, perception technology in autonomous driving has developed rapidly in recent years, enabling vehicles to accurately detect and interpret surrounding environment for safe and efficient navigation. To achieve accurate and robust perception capabilities, autonomous vehicles are often equipped with multiple sensors, making sensor fusion a crucial part of the perception system. Among these fused sensors, radars and cameras enable a complementary and cost-effective perception of the surrounding environment regardless of lighting and weather conditions. This review aims to provide a comprehensive guideline for radar-camera fusion, particularly concentrating on perception tasks related to object detection and semantic segmentation.Based on the principles of the radar and camera sensors, we delve into the data processing process and representations, followed by an in-depth analysis and summary of radar-camera fusion datasets. In the review of methodologies in radar-camera fusion, we address interrogative questions, including "why to fuse", "what to fuse", "where to fuse", "when to fuse", and "how to fuse", subsequently discussing various challenges and potential research directions within this domain. To ease the retrieval and comparison of datasets and fusion methods, we also provide an interactive website: <a class="link-external link-https" href="https://radar-camera-fusion.github.io" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Artificial Intelligence,Robotics

What problem does this paper attempt to address?

The paper primarily focuses on the issue of radar and camera fusion perception in autonomous driving, specifically concentrating on the two fundamental tasks of object detection and semantic segmentation. With the development of deep learning technology, the perception technology in autonomous driving has rapidly improved, enabling vehicles to accurately detect and interpret the surrounding environment for safe and efficient navigation. To achieve accurate and robust perception capabilities, autonomous vehicles are typically equipped with multiple sensors, making sensor fusion a crucial part of the perception system. Among these, radar and cameras can provide complementary and cost-effective environmental perception under various lighting and weather conditions.

The core of the paper is to provide a comprehensive guide specifically for the application of radar and camera fusion in autonomous driving, particularly in the tasks of object detection and semantic segmentation. Based on the basic principles of radar and camera sensors, the paper delves into the data processing procedures and their representations, and provides a detailed analysis and summary of radar-camera fusion datasets. Additionally, the paper discusses key issues in the fusion process, such as why to fuse, what to fuse, where to fuse, when to fuse, and how to fuse, and further explores various challenges and potential research directions in this field. To facilitate the retrieval and comparison of datasets and fusion methods, the authors also provide an interactive website.

In summary, the problem this paper attempts to address is how to effectively utilize the complementary advantages of radar and cameras in autonomous driving scenarios to improve the accuracy and robustness of object detection and semantic segmentation.