Abstract:Sensors have different perceptive abilities against environment. Sensor fusion plays a crucial role at achieving better perception by accumulating the information acquired at different times. But, the decision of the observation may conflict with each other due to usage of different algorithms, thresholds on processing algorithms, and different perceptive character of sensors. This study presents the late fusion method applied to outputs provided by deep learning models fed by camera and lidar sensor's measurement data. For camera sensor, a deep learning model as a multi-task network is proposed to multi-classify cars, motorcycles, bicycles, buses, trucks, and pedestrians under the category of dynamic traffic objects. In addition, color classified traffic lights and traffic signs with a capability of segmenting drivable area and detecting lane lines are classified under the category of static traffic objects. The proposed multi-network is trained and tested with the BDD100K dataset and benchmarked with publicly available multi-networks. The presented method is the second fastest multi-network reaching at 52 FPS runtime, ranked second based on the drivable area segmentation and lane line detection performance. For segmentation of dynamic objects, the network performance is increased by 22.45%, and considering mIoU overall performance increase is 3.96%. For a lidar sensor, a different modality is presented to detect objects. Two sensors' data are fused by proposed fusion algorithm, and results are tested and evaluated with the KITTI dataset. The proposed fusion methodology outperforms the stand-alone lidar methods about 3.58% and 3.63% on BEV and 3D detection MAP, respectively. Overall, benchmarking with two distinct fusion approaches illustrates the effectiveness of the proposed method.

Fast Road Detection by CNN-Based Camera–Lidar Fusion and Spherical Coordinate Transformation

ObjectFusion: an Object Detection and Segmentation Framework with RGB-D SLAM and Convolutional Neural Networks

Flexible Neural Network for Fast and Accurate Road Scene Perception.

SST-Calib: Simultaneous Spatial-Temporal Parameter Calibration between LIDAR and Camera

Road Scene Segmentation Via Fusing Camera and Lidar Data.

Multi-Stage Residual Fusion Network for LIDAR-Camera Road Detection

ResNet-based Surface Normal Estimator with Multilevel Fusion Approach with Adaptive Median Filter Region Growth Algorithm for Road Scene Segmentation

FastRoadSeg: Fast Monocular Road Segmentation Network

Real-Time 3D Object Detection From Point Cloud Through Foreground Segmentation

Road Segmentation with Image-LiDAR Data Fusion

Fast vehicle detection based on colored point cloud with bird's eye view representation

Late sensor fusion approach with a designed multi-segmentation network

Sparse-to-Dense LiDAR Point Generation by LiDAR-Camera Fusion for 3D Object Detection

Fast vehicle detection based on colored point cloud with bird’s eye view representation

3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-view Spatial Feature Fusion for 3D Object Detection

Closing the Calibration Gap: A Real-Time Multi-Modal Fusion Framework for 3D Semantic Segmentation

Real time object detection using LiDAR and camera fusion for autonomous driving

Fast and Robust Approaches for Lane Detection Using Multi‐camera Fusion in Complex Scenes

Complementary Features With Reasonable Receptive Field For Road Scene 3d Object Detection

Real-Time Vehicle Detection Framework Based on the Fusion of LiDAR and Camera

Revisiting Multi-modal 3D Semantic Segmentation in Real-world Autonomous Driving